[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:42:05 dist-test-slave-n1hw krb5kdc[21560](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:42:05 dist-test-slave-n1hw krb5kdc[21560](info): set up 2 sockets
Sep 26 16:42:05 dist-test-slave-n1hw krb5kdc[21560](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 16:42:06.033034 21553 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Sep 26 16:42:06 dist-test-slave-n1hw krb5kdc[21560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:42:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:42:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
I20260926 16:42:06.155171 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:41373
--webserver_interface=127.21.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:41331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:41373
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:06.261605 21573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:06.261931 21573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:06.262028 21573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:06.266016 21573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:42:06.266106 21573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:06.266145 21573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:06.266179 21573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:42:06.266211 21573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:42:06.271334 21573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:41331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:41373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:41373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:06.272346 21573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:06.273342 21573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:06.282352 21581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:06.282361 21578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:06.282420 21579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:06.282673 21573 server_base.cc:1034] running on GCE node
I20260926 16:42:06.283269 21573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:06.284292 21573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:06.285457 21573 hybrid_clock.cc:648] HybridClock initialized: now 1790440926285442 us; error 34 us; skew 500 ppm
Sep 26 16:42:06 dist-test-slave-n1hw krb5kdc[21560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440926, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:06.288228 21573 init.cc:377] Logged in from keytab as kudu/127.21.12.126@KRBTEST.COM (short username kudu)
I20260926 16:42:06.289041 21573 webserver.cc:460] Webserver started at http://127.21.12.126:40179/ using document root <none> and password file <none>
I20260926 16:42:06.289536 21573 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:06.289600 21573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:06.289813 21573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:06.291742 21573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "5b5727df84b84d749b8d2b0af5ac5b26"
format_stamp: "Formatted at 2026-09-26 16:42:06 on dist-test-slave-n1hw"
server_key: "93fc72963cd6de217aae25697eac94a1"
server_key_iv: "0856f79f7545232ed69ed439f19c4649"
server_key_version: "encryptionkey@0"
I20260926 16:42:06.292382 21573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "5b5727df84b84d749b8d2b0af5ac5b26"
format_stamp: "Formatted at 2026-09-26 16:42:06 on dist-test-slave-n1hw"
server_key: "93fc72963cd6de217aae25697eac94a1"
server_key_iv: "0856f79f7545232ed69ed439f19c4649"
server_key_version: "encryptionkey@0"
I20260926 16:42:06.296298 21573 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260926 16:42:06.298628 21588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:06.299821 21573 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:42:06.299926 21573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "5b5727df84b84d749b8d2b0af5ac5b26"
format_stamp: "Formatted at 2026-09-26 16:42:06 on dist-test-slave-n1hw"
server_key: "93fc72963cd6de217aae25697eac94a1"
server_key_iv: "0856f79f7545232ed69ed439f19c4649"
server_key_version: "encryptionkey@0"
I20260926 16:42:06.300068 21573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:06.317687 21573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:06.321264 21573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:06.321467 21573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:06.327863 21573 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.126:41373
I20260926 16:42:06.327963 21640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.126:41373 every 8 connection(s)
I20260926 16:42:06.328847 21573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:42:06.331017 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 21573
I20260926 16:42:06.331174 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:42:06.332048 21641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:06.332789 21553 external_mini_cluster.cc:1428] Setting key b9d658bc16fcf40b50840f435486be8b
I20260926 16:42:06.338778 21641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b5727df84b84d749b8d2b0af5ac5b26: Bootstrap starting.
I20260926 16:42:06.341454 21641 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b5727df84b84d749b8d2b0af5ac5b26: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:06.342224 21641 log.cc:824] T 00000000000000000000000000000000 P 5b5727df84b84d749b8d2b0af5ac5b26: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:06.344480 21641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b5727df84b84d749b8d2b0af5ac5b26: No bootstrap required, opened a new log
I20260926 16:42:06.349709 21641 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b5727df84b84d749b8d2b0af5ac5b26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5727df84b84d749b8d2b0af5ac5b26" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 41373 } }
I20260926 16:42:06.350045 21641 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b5727df84b84d749b8d2b0af5ac5b26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:06.350121 21641 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b5727df84b84d749b8d2b0af5ac5b26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b5727df84b84d749b8d2b0af5ac5b26, State: Initialized, Role: FOLLOWER
I20260926 16:42:06.350744 21641 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b5727df84b84d749b8d2b0af5ac5b26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5727df84b84d749b8d2b0af5ac5b26" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 41373 } }
I20260926 16:42:06.350924 21641 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b5727df84b84d749b8d2b0af5ac5b26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:42:06.350988 21641 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b5727df84b84d749b8d2b0af5ac5b26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:42:06.351104 21641 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b5727df84b84d749b8d2b0af5ac5b26 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:06.352408 21641 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b5727df84b84d749b8d2b0af5ac5b26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5727df84b84d749b8d2b0af5ac5b26" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 41373 } }
I20260926 16:42:06.352779 21641 leader_election.cc:302] T 00000000000000000000000000000000 P 5b5727df84b84d749b8d2b0af5ac5b26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b5727df84b84d749b8d2b0af5ac5b26; no voters:
I20260926 16:42:06.353149 21641 leader_election.cc:288] T 00000000000000000000000000000000 P 5b5727df84b84d749b8d2b0af5ac5b26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:42:06.353240 21646 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b5727df84b84d749b8d2b0af5ac5b26 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:06.353473 21646 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b5727df84b84d749b8d2b0af5ac5b26 [term 1 LEADER]: Becoming Leader. State: Replica: 5b5727df84b84d749b8d2b0af5ac5b26, State: Running, Role: LEADER
I20260926 16:42:06.353826 21646 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b5727df84b84d749b8d2b0af5ac5b26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5727df84b84d749b8d2b0af5ac5b26" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 41373 } }
I20260926 16:42:06.354643 21641 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b5727df84b84d749b8d2b0af5ac5b26 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:42:06.354919 21646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b5727df84b84d749b8d2b0af5ac5b26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b5727df84b84d749b8d2b0af5ac5b26. Latest consensus state: current_term: 1 leader_uuid: "5b5727df84b84d749b8d2b0af5ac5b26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5727df84b84d749b8d2b0af5ac5b26" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 41373 } } }
I20260926 16:42:06.355031 21646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b5727df84b84d749b8d2b0af5ac5b26 [sys.catalog]: This master's current role is: LEADER
Sep 26 16:42:06 dist-test-slave-n1hw krb5kdc[21560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:42:06.355405 21650 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:42:06.357725 21647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b5727df84b84d749b8d2b0af5ac5b26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b5727df84b84d749b8d2b0af5ac5b26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5727df84b84d749b8d2b0af5ac5b26" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 41373 } } }
I20260926 16:42:06.357882 21647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b5727df84b84d749b8d2b0af5ac5b26 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:06.359373 21650 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:42:06.362567 21643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:06.336368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60832 (local address 127.21.12.126:41373)
0926 16:42:06.336809 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:06.336827 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:06.337842 (+ 1015us) server_negotiation.cc:407] Connection header received
0926 16:42:06.341810 (+ 3968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:06.341826 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:06.342119 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:06.342442 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:06.345277 (+ 2835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:06.347086 (+ 1809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:06.349017 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:06.350229 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:06.350468 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:06.356369 (+ 5901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:06.356378 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:06.356395 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:06.356427 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:06.358754 (+ 2327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:06.359367 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:06.359373 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:06.359378 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:06.359571 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:06.360068 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:06.360073 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:06.360075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:06.360509 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:06.360710 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:06.361341 (+ 631us) server_negotiation.cc:299] Negotiation successful
0926 16:42:06.361644 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":125,"threads_started":1}
I20260926 16:42:06.365140 21650 catalog_manager.cc:1347] Generated new cluster ID: 1fa0433e83964c839cdb5bc4810eddbd
I20260926 16:42:06.365200 21650 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:42:06.381816 21650 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:42:06.382709 21650 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:42:06.386705 21650 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b5727df84b84d749b8d2b0af5ac5b26: Generated new TSK 0
I20260926 16:42:06.387341 21650 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab.
Entry for principal kudu/127.21.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab.
I20260926 16:42:06.421522 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.65:0
--local_ip_for_outbound_sockets=127.21.12.65
--webserver_interface=127.21.12.65
--webserver_port=0
--tserver_master_addrs=127.21.12.126:41373
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab
--principal=kudu/127.21.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:41331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:06.530812 21667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:06.531046 21667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:06.531090 21667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:06.535141 21667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:06.535192 21667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:06.535280 21667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.65
I20260926 16:42:06.540350 21667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:41331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab
--principal=kudu/127.21.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.65
--webserver_port=0
--tserver_master_addrs=127.21.12.126:41373
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.65
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:06.541311 21667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:06.542325 21667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:06.551613 21675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:06.551673 21673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:06.551597 21672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:06.551798 21667 server_base.cc:1034] running on GCE node
I20260926 16:42:06.552332 21667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:06.552953 21667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:06.554191 21667 hybrid_clock.cc:648] HybridClock initialized: now 1790440926554175 us; error 31 us; skew 500 ppm
Sep 26 16:42:06 dist-test-slave-n1hw krb5kdc[21560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440926, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:06.561093 21667 init.cc:377] Logged in from keytab as kudu/127.21.12.65@KRBTEST.COM (short username kudu)
I20260926 16:42:06.562063 21667 webserver.cc:460] Webserver started at http://127.21.12.65:40911/ using document root <none> and password file <none>
I20260926 16:42:06.562628 21667 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:06.562722 21667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:06.562947 21667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:06.565011 21667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-0/data/instance:
uuid: "9be76b70f3ff401980cd802ffd68c737"
format_stamp: "Formatted at 2026-09-26 16:42:06 on dist-test-slave-n1hw"
server_key: "451de6b86ba4af8ae78d041eafc174e0"
server_key_iv: "01fb37c79cf7194074dafabcc5d173bf"
server_key_version: "encryptionkey@0"
I20260926 16:42:06.565589 21667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance:
uuid: "9be76b70f3ff401980cd802ffd68c737"
format_stamp: "Formatted at 2026-09-26 16:42:06 on dist-test-slave-n1hw"
server_key: "451de6b86ba4af8ae78d041eafc174e0"
server_key_iv: "01fb37c79cf7194074dafabcc5d173bf"
server_key_version: "encryptionkey@0"
I20260926 16:42:06.569325 21667 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 16:42:06.571800 21682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:06.573132 21667 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:42:06.573278 21667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-0/wal
uuid: "9be76b70f3ff401980cd802ffd68c737"
format_stamp: "Formatted at 2026-09-26 16:42:06 on dist-test-slave-n1hw"
server_key: "451de6b86ba4af8ae78d041eafc174e0"
server_key_iv: "01fb37c79cf7194074dafabcc5d173bf"
server_key_version: "encryptionkey@0"
I20260926 16:42:06.573395 21667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:06.596632 21667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:06.599993 21667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:06.600205 21667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:06.600785 21667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:42:06.601586 21667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:06.601655 21667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:06.601724 21667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:06.601802 21667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:06.613507 21667 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.65:35859
I20260926 16:42:06.613557 21795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.65:35859 every 8 connection(s)
I20260926 16:42:06.614611 21667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
I20260926 16:42:06.618498 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 21667
I20260926 16:42:06.618607 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance
I20260926 16:42:06.618878 21553 external_mini_cluster.cc:1428] Setting key 6f37cc92418e85a0cda72e3485eb5eca
Sep 26 16:42:06 dist-test-slave-n1hw krb5kdc[21560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440926, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.12.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.66@KRBTEST.COM" created.
I20260926 16:42:06.632701 21643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:06.617131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:43091 (local address 127.21.12.126:41373)
0926 16:42:06.617291 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:06.617295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:06.618521 (+ 1226us) server_negotiation.cc:407] Connection header received
0926 16:42:06.619798 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:06.619801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:06.619868 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:06.619948 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:06.621500 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:06.622152 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:06.623986 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:06.624328 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:06.624452 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:06.627771 (+ 3319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:06.627777 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:06.627781 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:06.627811 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:06.629699 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:06.630431 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:06.630434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:06.630437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:06.630539 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:06.630946 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:06.630952 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:06.630957 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:06.631225 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:06.631338 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:06.632113 (+ 775us) server_negotiation.cc:299] Negotiation successful
0926 16:42:06.632311 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":63}
I20260926 16:42:06.633540 21798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:06.617314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:41373 (local address 127.21.12.65:43091)
0926 16:42:06.618151 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:06.618212 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:06.619452 (+ 1240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:06.620131 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:06.620143 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:06.620639 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:06.621340 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:06.621363 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:06.622319 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:06.622326 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:06.623815 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:06.623829 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:06.624507 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:06.624514 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:06.624759 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:06.625476 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:06.625499 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:06.627572 (+ 2073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:06.629929 (+ 2357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:06.629937 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:06.629953 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:06.630270 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:06.630644 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:06.630648 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:06.630650 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:06.630829 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:06.631338 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:06.631349 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:06.631768 (+ 419us) client_negotiation.cc:769] Sending connection context
0926 16:42:06.632018 (+ 250us) client_negotiation.cc:240] Negotiation successful
0926 16:42:06.632397 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":106,"threads_started":1}
I20260926 16:42:06.635424 21796 heartbeater.cc:347] Connected to a master server at 127.21.12.126:41373
I20260926 16:42:06.635757 21796 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:06.636361 21796 heartbeater.cc:511] Master 127.21.12.126:41373 requested a full tablet report, sending...
I20260926 16:42:06.638258 21600 ts_manager.cc:194] Registered new tserver with Master: 9be76b70f3ff401980cd802ffd68c737 (127.21.12.65:35859)
I20260926 16:42:06.639900 21600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.65@KRBTEST.COM'} at 127.21.12.65:43091
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab.
Entry for principal kudu/127.21.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab.
I20260926 16:42:06.649700 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.66:0
--local_ip_for_outbound_sockets=127.21.12.66
--webserver_interface=127.21.12.66
--webserver_port=0
--tserver_master_addrs=127.21.12.126:41373
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab
--principal=kudu/127.21.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:41331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:06.761189 21801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:06.761425 21801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:06.761508 21801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:06.765448 21801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:06.765502 21801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:06.765631 21801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.66
I20260926 16:42:06.770898 21801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:41331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab
--principal=kudu/127.21.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.66
--webserver_port=0
--tserver_master_addrs=127.21.12.126:41373
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.66
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:06.771875 21801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:06.772872 21801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:06.782119 21809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:06.782191 21806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:06.782143 21807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:06.782500 21801 server_base.cc:1034] running on GCE node
I20260926 16:42:06.782899 21801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:06.783558 21801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:06.784745 21801 hybrid_clock.cc:648] HybridClock initialized: now 1790440926784728 us; error 26 us; skew 500 ppm
Sep 26 16:42:06 dist-test-slave-n1hw krb5kdc[21560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440926, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:06.787627 21801 init.cc:377] Logged in from keytab as kudu/127.21.12.66@KRBTEST.COM (short username kudu)
I20260926 16:42:06.788628 21801 webserver.cc:460] Webserver started at http://127.21.12.66:32901/ using document root <none> and password file <none>
I20260926 16:42:06.789204 21801 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:06.789295 21801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:06.789525 21801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:06.791568 21801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-1/data/instance:
uuid: "610c6136fade45caa522d2a7e880fca5"
format_stamp: "Formatted at 2026-09-26 16:42:06 on dist-test-slave-n1hw"
server_key: "a1522caaaddc2987e3992db71568069b"
server_key_iv: "49c1d7e72339b3637c4f6194052a3717"
server_key_version: "encryptionkey@0"
I20260926 16:42:06.792135 21801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance:
uuid: "610c6136fade45caa522d2a7e880fca5"
format_stamp: "Formatted at 2026-09-26 16:42:06 on dist-test-slave-n1hw"
server_key: "a1522caaaddc2987e3992db71568069b"
server_key_iv: "49c1d7e72339b3637c4f6194052a3717"
server_key_version: "encryptionkey@0"
I20260926 16:42:06.796007 21801 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 16:42:06.798417 21816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:06.799631 21801 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:42:06.799760 21801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-1/wal
uuid: "610c6136fade45caa522d2a7e880fca5"
format_stamp: "Formatted at 2026-09-26 16:42:06 on dist-test-slave-n1hw"
server_key: "a1522caaaddc2987e3992db71568069b"
server_key_iv: "49c1d7e72339b3637c4f6194052a3717"
server_key_version: "encryptionkey@0"
I20260926 16:42:06.799866 21801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:06.823627 21801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:06.826630 21801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:06.826843 21801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:06.827445 21801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:42:06.828281 21801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:06.828354 21801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:06.828428 21801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:06.828472 21801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:06.837713 21801 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.66:34317
I20260926 16:42:06.837738 21929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.66:34317 every 8 connection(s)
I20260926 16:42:06.838605 21801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
I20260926 16:42:06.846928 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 21801
I20260926 16:42:06.847059 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance
I20260926 16:42:06.847360 21553 external_mini_cluster.cc:1428] Setting key 8b78068087f603adc9b3079d3f422cb1
Sep 26 16:42:06 dist-test-slave-n1hw krb5kdc[21560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440926, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.66@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:42:06.853101 21643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:06.840389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:53103 (local address 127.21.12.126:41373)
0926 16:42:06.840535 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:06.840538 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:06.841640 (+ 1102us) server_negotiation.cc:407] Connection header received
0926 16:42:06.842698 (+ 1058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:06.842701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:06.842748 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:06.842826 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:06.844102 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:06.844668 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:06.845857 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:06.846077 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:06.846161 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:06.848916 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:06.848920 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:06.848923 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:06.848949 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:06.850708 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:06.851461 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:06.851464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:06.851465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:06.851542 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:06.851937 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:06.851939 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:06.851940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:06.852134 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:06.852209 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:06.852794 (+ 585us) server_negotiation.cc:299] Negotiation successful
0926 16:42:06.852925 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 16:42:06.853948 21932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:06.840783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:41373 (local address 127.21.12.66:53103)
0926 16:42:06.841489 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:06.841533 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:06.842478 (+ 945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:06.843030 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:06.843037 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:06.843414 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:06.843929 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:06.843960 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:06.844806 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:06.844809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:06.845728 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:06.845738 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:06.846212 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:06.846214 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:06.846357 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:06.846930 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:06.846957 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:06.848724 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:06.850923 (+ 2199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:06.850935 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:06.850957 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:06.851354 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:06.851635 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:06.851640 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:06.851644 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:06.851838 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:06.852246 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:06.852251 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:06.852537 (+ 286us) client_negotiation.cc:769] Sending connection context
0926 16:42:06.852749 (+ 212us) client_negotiation.cc:240] Negotiation successful
0926 16:42:06.853076 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":116,"threads_started":1}
I20260926 16:42:06.855124 21930 heartbeater.cc:347] Connected to a master server at 127.21.12.126:41373
I20260926 16:42:06.855398 21930 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:06.855913 21930 heartbeater.cc:511] Master 127.21.12.126:41373 requested a full tablet report, sending...
I20260926 16:42:06.857057 21600 ts_manager.cc:194] Registered new tserver with Master: 610c6136fade45caa522d2a7e880fca5 (127.21.12.66:34317)
WARNING: no policy specified for kudu/127.21.12.67@KRBTEST.COM; defaulting to no policy
I20260926 16:42:06.857656 21600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.66@KRBTEST.COM'} at 127.21.12.66:53103
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab.
Entry for principal kudu/127.21.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab.
I20260926 16:42:06.876521 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.67:0
--local_ip_for_outbound_sockets=127.21.12.67
--webserver_interface=127.21.12.67
--webserver_port=0
--tserver_master_addrs=127.21.12.126:41373
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab
--principal=kudu/127.21.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:41331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:06.985502 21935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:06.985738 21935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:06.985839 21935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:06.989899 21935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:06.989951 21935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:06.990036 21935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.67
I20260926 16:42:06.995216 21935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:41331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab
--principal=kudu/127.21.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.67
--webserver_port=0
--tserver_master_addrs=127.21.12.126:41373
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.67
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:06.996208 21935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:06.997213 21935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:07.006745 21940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:07.006731 21943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:07.007022 21935 server_base.cc:1034] running on GCE node
W20260926 16:42:07.006755 21941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:07.007500 21935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:07.008040 21935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:07.009197 21935 hybrid_clock.cc:648] HybridClock initialized: now 1790440927009185 us; error 25 us; skew 500 ppm
Sep 26 16:42:07 dist-test-slave-n1hw krb5kdc[21560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440927, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:07.012286 21935 init.cc:377] Logged in from keytab as kudu/127.21.12.67@KRBTEST.COM (short username kudu)
I20260926 16:42:07.013204 21935 webserver.cc:460] Webserver started at http://127.21.12.67:41105/ using document root <none> and password file <none>
I20260926 16:42:07.013808 21935 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:07.013880 21935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:07.014130 21935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:07.016134 21935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-2/data/instance:
uuid: "707dec31b58c41038693c07cc73bfc64"
format_stamp: "Formatted at 2026-09-26 16:42:07 on dist-test-slave-n1hw"
server_key: "db489b74a225e7e77ea2e643e2502496"
server_key_iv: "021587d639ed4be414424017321ca526"
server_key_version: "encryptionkey@0"
I20260926 16:42:07.016696 21935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance:
uuid: "707dec31b58c41038693c07cc73bfc64"
format_stamp: "Formatted at 2026-09-26 16:42:07 on dist-test-slave-n1hw"
server_key: "db489b74a225e7e77ea2e643e2502496"
server_key_iv: "021587d639ed4be414424017321ca526"
server_key_version: "encryptionkey@0"
I20260926 16:42:07.020313 21935 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 16:42:07.022667 21950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:07.023850 21935 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:42:07.023980 21935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-2/wal
uuid: "707dec31b58c41038693c07cc73bfc64"
format_stamp: "Formatted at 2026-09-26 16:42:07 on dist-test-slave-n1hw"
server_key: "db489b74a225e7e77ea2e643e2502496"
server_key_iv: "021587d639ed4be414424017321ca526"
server_key_version: "encryptionkey@0"
I20260926 16:42:07.024091 21935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:07.043883 21935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:07.047621 21935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:07.047820 21935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:07.048439 21935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:42:07.049283 21935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:07.049355 21935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:07.049431 21935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:07.049463 21935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:07.059187 21935 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.67:41049
I20260926 16:42:07.059212 22063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.67:41049 every 8 connection(s)
I20260926 16:42:07.060042 21935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
I20260926 16:42:07.062446 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 21935
I20260926 16:42:07.062590 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance
I20260926 16:42:07.062882 21553 external_mini_cluster.cc:1428] Setting key f162b15e880fcdcd5488cc69c87a0ebc
Sep 26 16:42:07 dist-test-slave-n1hw krb5kdc[21560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440927, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.67@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:42:07.073607 21643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:07.061804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:55313 (local address 127.21.12.126:41373)
0926 16:42:07.061927 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:07.061931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:07.063135 (+ 1204us) server_negotiation.cc:407] Connection header received
0926 16:42:07.064169 (+ 1034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:07.064172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:07.064239 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:07.064320 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:07.065655 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:07.066238 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:07.067400 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:07.067646 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:07.067721 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:07.070207 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:07.070210 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:07.070213 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:07.070235 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:07.071775 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:07.072246 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:07.072249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:07.072250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:07.072327 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:07.072685 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:07.072687 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:07.072689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:07.072866 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:07.072944 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:07.073397 (+ 453us) server_negotiation.cc:299] Negotiation successful
0926 16:42:07.073488 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 16:42:07.074436 22066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:07.062170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:41373 (local address 127.21.12.67:55313)
0926 16:42:07.062948 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:07.063006 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:07.063904 (+ 898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:07.064527 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:07.064535 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:07.064895 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:07.065524 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:07.065539 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:07.066365 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:07.066368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:07.067287 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:07.067296 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:07.067778 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:07.067782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:07.067940 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:07.068507 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:07.068522 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:07.070059 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:07.071875 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:07.071879 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:07.071891 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:07.072155 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:07.072408 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:07.072411 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:07.072413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:07.072586 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:07.072953 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:07.072957 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:07.073196 (+ 239us) client_negotiation.cc:769] Sending connection context
0926 16:42:07.073350 (+ 154us) client_negotiation.cc:240] Negotiation successful
0926 16:42:07.073588 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":105,"threads_started":1}
I20260926 16:42:07.075554 22064 heartbeater.cc:347] Connected to a master server at 127.21.12.126:41373
I20260926 16:42:07.075812 22064 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:07.076370 22064 heartbeater.cc:511] Master 127.21.12.126:41373 requested a full tablet report, sending...
I20260926 16:42:07.077484 21600 ts_manager.cc:194] Registered new tserver with Master: 707dec31b58c41038693c07cc73bfc64 (127.21.12.67:41049)
I20260926 16:42:07.078207 21600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.67@KRBTEST.COM'} at 127.21.12.67:55313
I20260926 16:42:07.088770 21553 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 16:42:07 dist-test-slave-n1hw krb5kdc[21560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440927, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 16:42:07 dist-test-slave-n1hw krb5kdc[21560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440927, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.12.65@KRBTEST.COM
I20260926 16:42:07.113708 22071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:07.104537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41044 (local address 127.21.12.65:35859)
0926 16:42:07.104836 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:07.104850 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:07.104876 (+ 26us) server_negotiation.cc:407] Connection header received
0926 16:42:07.105050 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:07.105056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:07.105244 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:07.105365 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:07.106164 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:07.106987 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:07.107770 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:07.108129 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:07.108213 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:07.110511 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:07.110519 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:07.110531 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:07.110553 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:07.112122 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:07.112496 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:07.112500 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:07.112501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:07.112576 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:07.112812 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:07.112814 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:07.112816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:07.113036 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:07.113197 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:07.113341 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 16:42:07.113497 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":103,"threads_started":1}
W20260926 16:42:07.114743 21709 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41044
Sep 26 16:42:07 dist-test-slave-n1hw krb5kdc[21560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 16:42:07 dist-test-slave-n1hw krb5kdc[21560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.65@KRBTEST.COM
I20260926 16:42:07.140388 22071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:07.131871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41054 (local address 127.21.12.65:35859)
0926 16:42:07.132027 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:07.132032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:07.132186 (+ 154us) server_negotiation.cc:407] Connection header received
0926 16:42:07.132358 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:07.132366 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:07.132412 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:07.132501 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:07.133184 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:07.133703 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:07.134623 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:07.134861 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:07.134967 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:07.137400 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:07.137403 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:07.137405 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:07.137428 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:07.138747 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:07.139165 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:07.139170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:07.139172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:07.139254 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:07.139543 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:07.139547 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:07.139548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:07.139721 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:07.139860 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:07.140091 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 16:42:07.140220 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 16:42:07.141568 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 21667
I20260926 16:42:07.147243 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 21801
I20260926 16:42:07.154963 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 21935
I20260926 16:42:07.160177 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 21573
2026-09-26T16:42:07Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2217 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:42:07 dist-test-slave-n1hw krb5kdc[22078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:42:07 dist-test-slave-n1hw krb5kdc[22078](info): set up 2 sockets
Sep 26 16:42:07 dist-test-slave-n1hw krb5kdc[22078](info): commencing operation
krb5kdc: starting...
W20260926 16:42:08.195581 21553 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Sep 26 16:42:08 dist-test-slave-n1hw krb5kdc[22078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:42:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:42:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
I20260926 16:42:08.311175 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:41565
--webserver_interface=127.21.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:41193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:41565
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:08.419152 22091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:08.419488 22091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:08.419584 22091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:08.423643 22091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:42:08.423743 22091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:08.423786 22091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:08.423852 22091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:42:08.423895 22091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:42:08.429131 22091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:41193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:41565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:41565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:08.430207 22091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:08.431245 22091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:08.439908 22096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:08.439899 22099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:08.440121 22091 server_base.cc:1034] running on GCE node
W20260926 16:42:08.439898 22097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:08.440786 22091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:08.441946 22091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:08.443150 22091 hybrid_clock.cc:648] HybridClock initialized: now 1790440928443120 us; error 43 us; skew 500 ppm
Sep 26 16:42:08 dist-test-slave-n1hw krb5kdc[22078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440928, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:08.446161 22091 init.cc:377] Logged in from keytab as kudu/127.21.12.126@KRBTEST.COM (short username kudu)
I20260926 16:42:08.447165 22091 webserver.cc:460] Webserver started at http://127.21.12.126:46745/ using document root <none> and password file <none>
I20260926 16:42:08.447724 22091 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:08.447789 22091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:08.448040 22091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:08.450160 22091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "81550ea03c814703b03fc0820d8f76a0"
format_stamp: "Formatted at 2026-09-26 16:42:08 on dist-test-slave-n1hw"
server_key: "002fa33a23fca71d5d725dae3def5f6b"
server_key_iv: "169bee5dd4f06878ee6ce73a538d05ad"
server_key_version: "encryptionkey@0"
I20260926 16:42:08.450744 22091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "81550ea03c814703b03fc0820d8f76a0"
format_stamp: "Formatted at 2026-09-26 16:42:08 on dist-test-slave-n1hw"
server_key: "002fa33a23fca71d5d725dae3def5f6b"
server_key_iv: "169bee5dd4f06878ee6ce73a538d05ad"
server_key_version: "encryptionkey@0"
I20260926 16:42:08.454507 22091 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 16:42:08.456983 22106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:08.458319 22091 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 16:42:08.458470 22091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "81550ea03c814703b03fc0820d8f76a0"
format_stamp: "Formatted at 2026-09-26 16:42:08 on dist-test-slave-n1hw"
server_key: "002fa33a23fca71d5d725dae3def5f6b"
server_key_iv: "169bee5dd4f06878ee6ce73a538d05ad"
server_key_version: "encryptionkey@0"
I20260926 16:42:08.458583 22091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:08.475499 22091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:08.478473 22091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:08.478678 22091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:08.485841 22091 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.126:41565
I20260926 16:42:08.485868 22158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.126:41565 every 8 connection(s)
I20260926 16:42:08.486828 22091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:42:08.489645 22159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:08.495457 22159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81550ea03c814703b03fc0820d8f76a0: Bootstrap starting.
I20260926 16:42:08.496805 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 22091
I20260926 16:42:08.496896 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:42:08.497206 21553 external_mini_cluster.cc:1428] Setting key 2a05891009d68d377758778417c57541
I20260926 16:42:08.498009 22159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81550ea03c814703b03fc0820d8f76a0: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:08.498867 22159 log.cc:824] T 00000000000000000000000000000000 P 81550ea03c814703b03fc0820d8f76a0: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:08.500697 22159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81550ea03c814703b03fc0820d8f76a0: No bootstrap required, opened a new log
I20260926 16:42:08.504076 22159 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81550ea03c814703b03fc0820d8f76a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81550ea03c814703b03fc0820d8f76a0" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 41565 } }
I20260926 16:42:08.504276 22159 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81550ea03c814703b03fc0820d8f76a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:08.504341 22159 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81550ea03c814703b03fc0820d8f76a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81550ea03c814703b03fc0820d8f76a0, State: Initialized, Role: FOLLOWER
I20260926 16:42:08.504791 22159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81550ea03c814703b03fc0820d8f76a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81550ea03c814703b03fc0820d8f76a0" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 41565 } }
I20260926 16:42:08.504910 22159 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81550ea03c814703b03fc0820d8f76a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:42:08.504999 22159 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81550ea03c814703b03fc0820d8f76a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:42:08.505098 22159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81550ea03c814703b03fc0820d8f76a0 [term 0 FOLLOWER]: Advancing to term 1
Sep 26 16:42:08 dist-test-slave-n1hw krb5kdc[22078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:42:08.506273 22159 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81550ea03c814703b03fc0820d8f76a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81550ea03c814703b03fc0820d8f76a0" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 41565 } }
I20260926 16:42:08.506608 22159 leader_election.cc:302] T 00000000000000000000000000000000 P 81550ea03c814703b03fc0820d8f76a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81550ea03c814703b03fc0820d8f76a0; no voters:
I20260926 16:42:08.506896 22159 leader_election.cc:288] T 00000000000000000000000000000000 P 81550ea03c814703b03fc0820d8f76a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:42:08.507148 22164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81550ea03c814703b03fc0820d8f76a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:08.507360 22164 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81550ea03c814703b03fc0820d8f76a0 [term 1 LEADER]: Becoming Leader. State: Replica: 81550ea03c814703b03fc0820d8f76a0, State: Running, Role: LEADER
I20260926 16:42:08.507694 22164 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81550ea03c814703b03fc0820d8f76a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81550ea03c814703b03fc0820d8f76a0" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 41565 } }
I20260926 16:42:08.508179 22159 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81550ea03c814703b03fc0820d8f76a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:42:08.509557 22165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81550ea03c814703b03fc0820d8f76a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81550ea03c814703b03fc0820d8f76a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81550ea03c814703b03fc0820d8f76a0" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 41565 } } }
I20260926 16:42:08.509541 22166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81550ea03c814703b03fc0820d8f76a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81550ea03c814703b03fc0820d8f76a0. Latest consensus state: current_term: 1 leader_uuid: "81550ea03c814703b03fc0820d8f76a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81550ea03c814703b03fc0820d8f76a0" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 41565 } } }
I20260926 16:42:08.509812 22165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81550ea03c814703b03fc0820d8f76a0 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:08.509825 22166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81550ea03c814703b03fc0820d8f76a0 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:08.510561 22176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:42:08.512591 22162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:08.498362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40816 (local address 127.21.12.126:41565)
0926 16:42:08.498795 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:08.498809 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:08.498842 (+ 33us) server_negotiation.cc:407] Connection header received
0926 16:42:08.499522 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:08.499539 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:08.499849 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:08.500183 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:08.500995 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:08.502067 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:08.502912 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:08.503390 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:08.503575 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:08.506499 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:08.506505 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:08.506527 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:08.506553 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:08.508881 (+ 2328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:08.509520 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:08.509528 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:08.509536 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:08.509722 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:08.510127 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:08.510131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:08.510133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:08.510661 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:08.510867 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:08.511244 (+ 377us) server_negotiation.cc:299] Negotiation successful
0926 16:42:08.511565 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":130,"threads_started":1}
I20260926 16:42:08.513429 22176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:42:08.519156 22176 catalog_manager.cc:1347] Generated new cluster ID: 4cdf88f22c9b423eac03d9b99cb81b42
I20260926 16:42:08.519256 22176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:42:08.547894 22176 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:42:08.548935 22176 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:42:08.556645 22176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81550ea03c814703b03fc0820d8f76a0: Generated new TSK 0
I20260926 16:42:08.557331 22176 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab.
Entry for principal kudu/127.21.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab.
I20260926 16:42:08.592958 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.65:0
--local_ip_for_outbound_sockets=127.21.12.65
--webserver_interface=127.21.12.65
--webserver_port=0
--tserver_master_addrs=127.21.12.126:41565
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab
--principal=kudu/127.21.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:41193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:08.698912 22185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:08.699136 22185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:08.699177 22185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:08.703151 22185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:08.703199 22185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:08.703256 22185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.65
I20260926 16:42:08.709424 22185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:41193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab
--principal=kudu/127.21.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.65
--webserver_port=0
--tserver_master_addrs=127.21.12.126:41565
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.65
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:08.710426 22185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:08.711371 22185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:08.720280 22193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:08.720397 22190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:08.720287 22191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:08.720646 22185 server_base.cc:1034] running on GCE node
I20260926 16:42:08.721005 22185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:08.721624 22185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:08.722784 22185 hybrid_clock.cc:648] HybridClock initialized: now 1790440928722761 us; error 41 us; skew 500 ppm
Sep 26 16:42:08 dist-test-slave-n1hw krb5kdc[22078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440928, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:08.725548 22185 init.cc:377] Logged in from keytab as kudu/127.21.12.65@KRBTEST.COM (short username kudu)
I20260926 16:42:08.726392 22185 webserver.cc:460] Webserver started at http://127.21.12.65:38333/ using document root <none> and password file <none>
I20260926 16:42:08.726926 22185 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:08.726982 22185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:08.727192 22185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:08.729099 22185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-0/data/instance:
uuid: "870a8388be904e30b84c8f53ead30f25"
format_stamp: "Formatted at 2026-09-26 16:42:08 on dist-test-slave-n1hw"
server_key: "3f46dd43c9c8b57e6512a4073fcc7cce"
server_key_iv: "cad2d26f03116e5e4e6b2f38e261ab06"
server_key_version: "encryptionkey@0"
I20260926 16:42:08.729648 22185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance:
uuid: "870a8388be904e30b84c8f53ead30f25"
format_stamp: "Formatted at 2026-09-26 16:42:08 on dist-test-slave-n1hw"
server_key: "3f46dd43c9c8b57e6512a4073fcc7cce"
server_key_iv: "cad2d26f03116e5e4e6b2f38e261ab06"
server_key_version: "encryptionkey@0"
I20260926 16:42:08.733218 22185 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260926 16:42:08.735477 22200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:08.736558 22185 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 16:42:08.736698 22185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-0/wal
uuid: "870a8388be904e30b84c8f53ead30f25"
format_stamp: "Formatted at 2026-09-26 16:42:08 on dist-test-slave-n1hw"
server_key: "3f46dd43c9c8b57e6512a4073fcc7cce"
server_key_iv: "cad2d26f03116e5e4e6b2f38e261ab06"
server_key_version: "encryptionkey@0"
I20260926 16:42:08.736804 22185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:08.751195 22185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:08.753993 22185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:08.754179 22185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:08.754722 22185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:42:08.755479 22185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:08.755549 22185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:08.755618 22185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:08.755667 22185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:08.764048 22185 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.65:44291
I20260926 16:42:08.764081 22313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.65:44291 every 8 connection(s)
I20260926 16:42:08.764863 22185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
I20260926 16:42:08.768783 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 22185
I20260926 16:42:08.768880 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance
I20260926 16:42:08.769115 21553 external_mini_cluster.cc:1428] Setting key 156cf769e3e29f544f388e2d15e656e4
Sep 26 16:42:08 dist-test-slave-n1hw krb5kdc[22078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440928, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.12.66@KRBTEST.COM; defaulting to no policy
I20260926 16:42:08.778798 22162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:08.766614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:37337 (local address 127.21.12.126:41565)
0926 16:42:08.766769 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:08.766774 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:08.767539 (+ 765us) server_negotiation.cc:407] Connection header received
0926 16:42:08.768775 (+ 1236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:08.768779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:08.768835 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:08.768915 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:08.770459 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:08.771223 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:08.772387 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:08.772627 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:08.772739 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:08.775276 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:08.775279 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:08.775281 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:08.775302 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:08.776743 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:08.777260 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:08.777265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:08.777267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:08.777345 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:08.777693 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:08.777697 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:08.777699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:08.777928 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:08.778037 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:08.778507 (+ 470us) server_negotiation.cc:299] Negotiation successful
0926 16:42:08.778630 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 16:42:08.779528 22316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:08.766981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:41565 (local address 127.21.12.65:37337)
0926 16:42:08.767395 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:08.767434 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:08.768574 (+ 1140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:08.769195 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:08.769204 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:08.769678 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:08.770328 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:08.770347 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:08.771354 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:08.771357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:08.772261 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:08.772270 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:08.772773 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:08.772777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:08.772944 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:08.773617 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:08.773633 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:08.775107 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:08.776868 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:08.776873 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:08.776887 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:08.777157 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:08.777443 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:08.777448 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:08.777450 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:08.777609 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:08.778043 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:08.778052 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:08.778275 (+ 223us) client_negotiation.cc:769] Sending connection context
0926 16:42:08.778466 (+ 191us) client_negotiation.cc:240] Negotiation successful
0926 16:42:08.778761 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":115,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.66@KRBTEST.COM" created.
I20260926 16:42:08.780939 22314 heartbeater.cc:347] Connected to a master server at 127.21.12.126:41565
I20260926 16:42:08.781201 22314 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:08.781729 22314 heartbeater.cc:511] Master 127.21.12.126:41565 requested a full tablet report, sending...
I20260926 16:42:08.783403 22123 ts_manager.cc:194] Registered new tserver with Master: 870a8388be904e30b84c8f53ead30f25 (127.21.12.65:44291)
I20260926 16:42:08.784509 22123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.65@KRBTEST.COM'} at 127.21.12.65:37337
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab.
Entry for principal kudu/127.21.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab.
I20260926 16:42:08.796875 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.66:0
--local_ip_for_outbound_sockets=127.21.12.66
--webserver_interface=127.21.12.66
--webserver_port=0
--tserver_master_addrs=127.21.12.126:41565
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab
--principal=kudu/127.21.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:41193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:08.902360 22319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:08.902585 22319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:08.902657 22319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:08.906579 22319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:08.906627 22319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:08.906709 22319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.66
I20260926 16:42:08.912065 22319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:41193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab
--principal=kudu/127.21.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.66
--webserver_port=0
--tserver_master_addrs=127.21.12.126:41565
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.66
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:08.912967 22319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:08.913935 22319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:08.923192 22324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:08.923236 22327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:08.923244 22325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:08.923489 22319 server_base.cc:1034] running on GCE node
I20260926 16:42:08.923854 22319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:08.924428 22319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:08.925611 22319 hybrid_clock.cc:648] HybridClock initialized: now 1790440928925589 us; error 33 us; skew 500 ppm
Sep 26 16:42:08 dist-test-slave-n1hw krb5kdc[22078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440928, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:08.928272 22319 init.cc:377] Logged in from keytab as kudu/127.21.12.66@KRBTEST.COM (short username kudu)
I20260926 16:42:08.929141 22319 webserver.cc:460] Webserver started at http://127.21.12.66:46629/ using document root <none> and password file <none>
I20260926 16:42:08.929683 22319 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:08.929796 22319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:08.930020 22319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:08.931944 22319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-1/data/instance:
uuid: "62a20e37a2ea452aa95e008b391f399c"
format_stamp: "Formatted at 2026-09-26 16:42:08 on dist-test-slave-n1hw"
server_key: "8b22a424cbf0984605caaffa0a603836"
server_key_iv: "9834efbd3b7c929c4cc040bc1a1cfa8f"
server_key_version: "encryptionkey@0"
I20260926 16:42:08.932492 22319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance:
uuid: "62a20e37a2ea452aa95e008b391f399c"
format_stamp: "Formatted at 2026-09-26 16:42:08 on dist-test-slave-n1hw"
server_key: "8b22a424cbf0984605caaffa0a603836"
server_key_iv: "9834efbd3b7c929c4cc040bc1a1cfa8f"
server_key_version: "encryptionkey@0"
I20260926 16:42:08.935981 22319 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260926 16:42:08.938216 22334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:08.939333 22319 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 16:42:08.939461 22319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-1/wal
uuid: "62a20e37a2ea452aa95e008b391f399c"
format_stamp: "Formatted at 2026-09-26 16:42:08 on dist-test-slave-n1hw"
server_key: "8b22a424cbf0984605caaffa0a603836"
server_key_iv: "9834efbd3b7c929c4cc040bc1a1cfa8f"
server_key_version: "encryptionkey@0"
I20260926 16:42:08.939563 22319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:08.963737 22319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:08.966730 22319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:08.966928 22319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:08.967492 22319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:42:08.968290 22319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:08.968369 22319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:08.968458 22319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:08.968508 22319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:08.977260 22319 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.66:40873
I20260926 16:42:08.977284 22447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.66:40873 every 8 connection(s)
I20260926 16:42:08.978140 22319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
I20260926 16:42:08.982679 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 22319
I20260926 16:42:08.982798 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance
I20260926 16:42:08.983088 21553 external_mini_cluster.cc:1428] Setting key a1088e0ee1dab26c2fe085d0204a121c
Sep 26 16:42:08 dist-test-slave-n1hw krb5kdc[22078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440928, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.66@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:42:08.992447 22162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:08.979875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:34711 (local address 127.21.12.126:41565)
0926 16:42:08.980002 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:08.980005 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:08.980886 (+ 881us) server_negotiation.cc:407] Connection header received
0926 16:42:08.982272 (+ 1386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:08.982282 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:08.982329 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:08.982407 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:08.983914 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:08.984444 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:08.985572 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:08.985873 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:08.985988 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:08.988409 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:08.988416 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:08.988422 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:08.988455 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:08.990292 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:08.990785 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:08.990790 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:08.990792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:08.990884 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:08.991267 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:08.991271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:08.991273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:08.991520 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:08.991637 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:08.992130 (+ 493us) server_negotiation.cc:299] Negotiation successful
0926 16:42:08.992251 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
WARNING: no policy specified for kudu/127.21.12.67@KRBTEST.COM; defaulting to no policy
I20260926 16:42:08.993207 22450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:08.980235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:41565 (local address 127.21.12.66:34711)
0926 16:42:08.980713 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:08.980767 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:08.982047 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:08.982726 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:08.982738 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:08.983222 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:08.983767 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:08.983781 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:08.984585 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:08.984589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:08.985444 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:08.985453 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:08.986022 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:08.986026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:08.986199 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:08.986796 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:08.986813 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:08.988238 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:08.990417 (+ 2179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:08.990422 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:08.990433 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:08.990679 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:08.991033 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:08.991036 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:08.991037 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:08.991164 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:08.991654 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:08.991659 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:08.991873 (+ 214us) client_negotiation.cc:769] Sending connection context
0926 16:42:08.992031 (+ 158us) client_negotiation.cc:240] Negotiation successful
0926 16:42:08.992296 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":112,"threads_started":1}
I20260926 16:42:08.994426 22448 heartbeater.cc:347] Connected to a master server at 127.21.12.126:41565
I20260926 16:42:08.994666 22448 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:08.995127 22448 heartbeater.cc:511] Master 127.21.12.126:41565 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.67@KRBTEST.COM" created.
I20260926 16:42:08.996173 22123 ts_manager.cc:194] Registered new tserver with Master: 62a20e37a2ea452aa95e008b391f399c (127.21.12.66:40873)
I20260926 16:42:08.996743 22123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.66@KRBTEST.COM'} at 127.21.12.66:34711
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab.
Entry for principal kudu/127.21.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab.
I20260926 16:42:09.012468 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.67:0
--local_ip_for_outbound_sockets=127.21.12.67
--webserver_interface=127.21.12.67
--webserver_port=0
--tserver_master_addrs=127.21.12.126:41565
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab
--principal=kudu/127.21.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:41193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:09.119845 22453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:09.120098 22453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:09.120167 22453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:09.124147 22453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:09.124222 22453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:09.124331 22453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.67
I20260926 16:42:09.129434 22453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:41193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab
--principal=kudu/127.21.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.67
--webserver_port=0
--tserver_master_addrs=127.21.12.126:41565
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.67
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:09.130672 22453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:09.131773 22453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:09.141088 22458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:09.141222 22461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:09.141119 22459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:09.141391 22453 server_base.cc:1034] running on GCE node
I20260926 16:42:09.141811 22453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:09.142446 22453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:09.143666 22453 hybrid_clock.cc:648] HybridClock initialized: now 1790440929143659 us; error 29 us; skew 500 ppm
Sep 26 16:42:09 dist-test-slave-n1hw krb5kdc[22078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440929, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:09.146396 22453 init.cc:377] Logged in from keytab as kudu/127.21.12.67@KRBTEST.COM (short username kudu)
I20260926 16:42:09.147221 22453 webserver.cc:460] Webserver started at http://127.21.12.67:41761/ using document root <none> and password file <none>
I20260926 16:42:09.147763 22453 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:09.147820 22453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:09.148032 22453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:09.150053 22453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-2/data/instance:
uuid: "f9dc15d8230743249730a768672a33a3"
format_stamp: "Formatted at 2026-09-26 16:42:09 on dist-test-slave-n1hw"
server_key: "ee0b49380723c99bbf969c2a31b01823"
server_key_iv: "9781d01f0dec28a3fcc408535ba1d28d"
server_key_version: "encryptionkey@0"
I20260926 16:42:09.150593 22453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance:
uuid: "f9dc15d8230743249730a768672a33a3"
format_stamp: "Formatted at 2026-09-26 16:42:09 on dist-test-slave-n1hw"
server_key: "ee0b49380723c99bbf969c2a31b01823"
server_key_iv: "9781d01f0dec28a3fcc408535ba1d28d"
server_key_version: "encryptionkey@0"
I20260926 16:42:09.154063 22453 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 16:42:09.156262 22468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:09.157387 22453 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:42:09.157521 22453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-2/wal
uuid: "f9dc15d8230743249730a768672a33a3"
format_stamp: "Formatted at 2026-09-26 16:42:09 on dist-test-slave-n1hw"
server_key: "ee0b49380723c99bbf969c2a31b01823"
server_key_iv: "9781d01f0dec28a3fcc408535ba1d28d"
server_key_version: "encryptionkey@0"
I20260926 16:42:09.157641 22453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:09.168306 22453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:09.171226 22453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:09.171418 22453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:09.171968 22453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:42:09.172747 22453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:09.172817 22453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:09.172887 22453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:09.172930 22453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:09.181644 22453 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.67:44535
I20260926 16:42:09.181666 22581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.67:44535 every 8 connection(s)
I20260926 16:42:09.182509 22453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
I20260926 16:42:09.187630 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 22453
I20260926 16:42:09.187749 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance
I20260926 16:42:09.188043 21553 external_mini_cluster.cc:1428] Setting key c42163122d09e3b195bcb6001b9a3209
Sep 26 16:42:09 dist-test-slave-n1hw krb5kdc[22078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440929, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.67@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:42:09.196358 22162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:09.184402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:46715 (local address 127.21.12.126:41565)
0926 16:42:09.184536 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:09.184540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:09.185290 (+ 750us) server_negotiation.cc:407] Connection header received
0926 16:42:09.186622 (+ 1332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:09.186625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:09.186683 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:09.186784 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:09.188074 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:09.188649 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:09.190301 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:09.190522 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:09.190622 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:09.192867 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:09.192870 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:09.192871 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:09.192894 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:09.194475 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:09.194917 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:09.194920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:09.194921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:09.194996 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:09.195407 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:09.195410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:09.195411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:09.195632 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:09.195720 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:09.196154 (+ 434us) server_negotiation.cc:299] Negotiation successful
0926 16:42:09.196247 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 16:42:09.197153 22584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:09.184752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:41565 (local address 127.21.12.67:46715)
0926 16:42:09.185156 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:09.185195 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:09.186410 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:09.186944 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:09.186951 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:09.187280 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:09.187917 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:09.187937 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:09.188773 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:09.188777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:09.190168 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:09.190177 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:09.190650 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:09.190653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:09.190778 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:09.191353 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:09.191368 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:09.192726 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:09.194575 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:09.194580 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:09.194590 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:09.194832 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:09.195104 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:09.195107 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:09.195108 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:09.195273 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:09.195735 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:09.195740 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:09.195950 (+ 210us) client_negotiation.cc:769] Sending connection context
0926 16:42:09.196101 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 16:42:09.196368 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":110,"threads_started":1}
I20260926 16:42:09.198319 22582 heartbeater.cc:347] Connected to a master server at 127.21.12.126:41565
I20260926 16:42:09.198513 22582 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:09.198930 22582 heartbeater.cc:511] Master 127.21.12.126:41565 requested a full tablet report, sending...
I20260926 16:42:09.199841 22123 ts_manager.cc:194] Registered new tserver with Master: f9dc15d8230743249730a768672a33a3 (127.21.12.67:44535)
I20260926 16:42:09.200423 22123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.67@KRBTEST.COM'} at 127.21.12.67:46715
I20260926 16:42:09.201802 21553 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:42:09.212308 22162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:09.205026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40820 (local address 127.21.12.126:41565)
0926 16:42:09.205164 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:09.205167 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:09.205325 (+ 158us) server_negotiation.cc:407] Connection header received
0926 16:42:09.205405 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:09.205407 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:09.205440 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:09.205502 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:09.206215 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:09.206704 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:09.207447 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:09.207709 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:09.207802 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:09.209170 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:09.209173 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:09.209175 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:09.209195 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:09.210692 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:09.211098 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:09.211102 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:09.211104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:09.211187 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:09.211458 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:09.211462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:09.211463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:09.211673 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:09.211815 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:09.212041 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 16:42:09.212159 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 16:42:09.216566 22122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40820:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:42:09.219542 22122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:42:09.239198 22594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:09.232896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:44291 (local address 127.0.0.1:43980)
0926 16:42:09.233850 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:09.233900 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:09.234027 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:09.234742 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:09.234749 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:09.234772 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:09.234918 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:09.234926 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:09.236514 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:09.236517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:09.237513 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:09.237523 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:09.238754 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:09.238757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:09.238831 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:09.238860 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 16:42:09.238941 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 16:42:09.239034 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"mutex_wait_us":44,"thread_start_us":67,"threads_started":1}
I20260926 16:42:09.239292 22595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:09.233010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43980 (local address 127.21.12.65:44291)
0926 16:42:09.233445 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:09.233464 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:09.234212 (+ 748us) server_negotiation.cc:407] Connection header received
0926 16:42:09.234372 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:09.234377 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:09.234589 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:09.234744 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:09.235196 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:09.236372 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:09.237886 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:09.238639 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:09.238748 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:09.238880 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:09.239044 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 16:42:09.239145 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":144,"threads_started":1}
I20260926 16:42:09.239532 22596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:09.232966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:40873 (local address 127.0.0.1:45176)
0926 16:42:09.234380 (+ 1414us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:09.234394 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:09.234483 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:09.235096 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:09.235102 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:09.235123 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:09.235220 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:09.235225 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:09.236177 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:09.236182 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:09.238073 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:09.238084 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:09.239131 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:09.239135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:09.239223 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:09.239255 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 16:42:09.239333 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 16:42:09.239406 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1355,"mutex_wait_us":436,"thread_start_us":90,"threads_started":1}
I20260926 16:42:09.239892 22597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:09.233577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45176 (local address 127.21.12.66:40873)
0926 16:42:09.234077 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:09.234090 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:09.234484 (+ 394us) server_negotiation.cc:407] Connection header received
0926 16:42:09.234572 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:09.234575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:09.234738 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:09.234878 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:09.235330 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:09.236052 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:09.238196 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:09.239013 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:09.239117 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:09.239239 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:09.239620 (+ 381us) server_negotiation.cc:299] Negotiation successful
0926 16:42:09.239728 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":152,"threads_started":1}
I20260926 16:42:09.240651 22593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:09.233545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.67:44535 (local address 127.0.0.1:41492)
0926 16:42:09.234668 (+ 1123us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:09.234684 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:09.234758 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:09.235376 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:09.235379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:09.235395 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:09.235477 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:09.235483 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:09.236745 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:09.236748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:09.238073 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:09.238084 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:09.240268 (+ 2184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:09.240271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:09.240363 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:09.240386 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 16:42:09.240461 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 16:42:09.240524 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1046,"mutex_wait_us":42,"thread_start_us":101,"threads_started":1}
I20260926 16:42:09.240749 22598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:09.233988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41492 (local address 127.21.12.67:44535)
0926 16:42:09.234342 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:09.234350 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:09.234846 (+ 496us) server_negotiation.cc:407] Connection header received
0926 16:42:09.234924 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:09.234927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:09.235091 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:09.235231 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:09.235598 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:09.236647 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:09.239295 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:09.240151 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:09.240253 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:09.240374 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:09.240505 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 16:42:09.240598 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":146,"threads_started":1}
I20260926 16:42:09.242686 22248 tablet_service.cc:1467] Processing CreateTablet for tablet d7671ff7b7054faeb3a6208f0d264c50 (DEFAULT_TABLE table=test-table [id=018bb7823ed64872b88370023fa90210]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:09.243201 22516 tablet_service.cc:1467] Processing CreateTablet for tablet d7671ff7b7054faeb3a6208f0d264c50 (DEFAULT_TABLE table=test-table [id=018bb7823ed64872b88370023fa90210]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:09.243677 22248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7671ff7b7054faeb3a6208f0d264c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:09.244163 22516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7671ff7b7054faeb3a6208f0d264c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:09.250612 22600 tablet_bootstrap.cc:492] T d7671ff7b7054faeb3a6208f0d264c50 P f9dc15d8230743249730a768672a33a3: Bootstrap starting.
I20260926 16:42:09.250626 22599 tablet_bootstrap.cc:492] T d7671ff7b7054faeb3a6208f0d264c50 P 870a8388be904e30b84c8f53ead30f25: Bootstrap starting.
I20260926 16:42:09.250267 22382 tablet_service.cc:1467] Processing CreateTablet for tablet d7671ff7b7054faeb3a6208f0d264c50 (DEFAULT_TABLE table=test-table [id=018bb7823ed64872b88370023fa90210]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:09.251259 22382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7671ff7b7054faeb3a6208f0d264c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:09.252367 22599 tablet_bootstrap.cc:654] T d7671ff7b7054faeb3a6208f0d264c50 P 870a8388be904e30b84c8f53ead30f25: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:09.252637 22600 tablet_bootstrap.cc:654] T d7671ff7b7054faeb3a6208f0d264c50 P f9dc15d8230743249730a768672a33a3: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:09.253300 22599 log.cc:824] T d7671ff7b7054faeb3a6208f0d264c50 P 870a8388be904e30b84c8f53ead30f25: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:09.253846 22600 log.cc:824] T d7671ff7b7054faeb3a6208f0d264c50 P f9dc15d8230743249730a768672a33a3: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:09.255172 22599 tablet_bootstrap.cc:492] T d7671ff7b7054faeb3a6208f0d264c50 P 870a8388be904e30b84c8f53ead30f25: No bootstrap required, opened a new log
I20260926 16:42:09.255357 22599 ts_tablet_manager.cc:1397] T d7671ff7b7054faeb3a6208f0d264c50 P 870a8388be904e30b84c8f53ead30f25: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 16:42:09.255738 22600 tablet_bootstrap.cc:492] T d7671ff7b7054faeb3a6208f0d264c50 P f9dc15d8230743249730a768672a33a3: No bootstrap required, opened a new log
I20260926 16:42:09.255910 22600 ts_tablet_manager.cc:1397] T d7671ff7b7054faeb3a6208f0d264c50 P f9dc15d8230743249730a768672a33a3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 16:42:09.257620 22603 tablet_bootstrap.cc:492] T d7671ff7b7054faeb3a6208f0d264c50 P 62a20e37a2ea452aa95e008b391f399c: Bootstrap starting.
I20260926 16:42:09.259198 22599 raft_consensus.cc:348] T d7671ff7b7054faeb3a6208f0d264c50 P 870a8388be904e30b84c8f53ead30f25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870a8388be904e30b84c8f53ead30f25" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 44291 } } peers { permanent_uuid: "62a20e37a2ea452aa95e008b391f399c" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40873 } } peers { permanent_uuid: "f9dc15d8230743249730a768672a33a3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 44535 } }
I20260926 16:42:09.259451 22599 raft_consensus.cc:374] T d7671ff7b7054faeb3a6208f0d264c50 P 870a8388be904e30b84c8f53ead30f25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:09.259548 22599 raft_consensus.cc:729] T d7671ff7b7054faeb3a6208f0d264c50 P 870a8388be904e30b84c8f53ead30f25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 870a8388be904e30b84c8f53ead30f25, State: Initialized, Role: FOLLOWER
I20260926 16:42:09.259461 22600 raft_consensus.cc:348] T d7671ff7b7054faeb3a6208f0d264c50 P f9dc15d8230743249730a768672a33a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870a8388be904e30b84c8f53ead30f25" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 44291 } } peers { permanent_uuid: "62a20e37a2ea452aa95e008b391f399c" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40873 } } peers { permanent_uuid: "f9dc15d8230743249730a768672a33a3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 44535 } }
I20260926 16:42:09.259722 22600 raft_consensus.cc:374] T d7671ff7b7054faeb3a6208f0d264c50 P f9dc15d8230743249730a768672a33a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:09.259781 22600 raft_consensus.cc:729] T d7671ff7b7054faeb3a6208f0d264c50 P f9dc15d8230743249730a768672a33a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9dc15d8230743249730a768672a33a3, State: Initialized, Role: FOLLOWER
I20260926 16:42:09.259845 22603 tablet_bootstrap.cc:654] T d7671ff7b7054faeb3a6208f0d264c50 P 62a20e37a2ea452aa95e008b391f399c: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:09.260106 22599 consensus_queue.cc:260] T d7671ff7b7054faeb3a6208f0d264c50 P 870a8388be904e30b84c8f53ead30f25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870a8388be904e30b84c8f53ead30f25" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 44291 } } peers { permanent_uuid: "62a20e37a2ea452aa95e008b391f399c" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40873 } } peers { permanent_uuid: "f9dc15d8230743249730a768672a33a3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 44535 } }
I20260926 16:42:09.260179 22600 consensus_queue.cc:260] T d7671ff7b7054faeb3a6208f0d264c50 P f9dc15d8230743249730a768672a33a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870a8388be904e30b84c8f53ead30f25" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 44291 } } peers { permanent_uuid: "62a20e37a2ea452aa95e008b391f399c" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40873 } } peers { permanent_uuid: "f9dc15d8230743249730a768672a33a3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 44535 } }
I20260926 16:42:09.260689 22603 log.cc:824] T d7671ff7b7054faeb3a6208f0d264c50 P 62a20e37a2ea452aa95e008b391f399c: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:09.260773 22314 heartbeater.cc:503] Master 127.21.12.126:41565 was elected leader, sending a full tablet report...
I20260926 16:42:09.261116 22599 ts_tablet_manager.cc:1428] T d7671ff7b7054faeb3a6208f0d264c50 P 870a8388be904e30b84c8f53ead30f25: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 16:42:09.261107 22600 ts_tablet_manager.cc:1428] T d7671ff7b7054faeb3a6208f0d264c50 P f9dc15d8230743249730a768672a33a3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 16:42:09.261438 22582 heartbeater.cc:503] Master 127.21.12.126:41565 was elected leader, sending a full tablet report...
I20260926 16:42:09.263473 22603 tablet_bootstrap.cc:492] T d7671ff7b7054faeb3a6208f0d264c50 P 62a20e37a2ea452aa95e008b391f399c: No bootstrap required, opened a new log
I20260926 16:42:09.263655 22603 ts_tablet_manager.cc:1397] T d7671ff7b7054faeb3a6208f0d264c50 P 62a20e37a2ea452aa95e008b391f399c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
W20260926 16:42:09.265883 22315 tablet.cc:2369] T d7671ff7b7054faeb3a6208f0d264c50 P 870a8388be904e30b84c8f53ead30f25: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:42:09.267117 22603 raft_consensus.cc:348] T d7671ff7b7054faeb3a6208f0d264c50 P 62a20e37a2ea452aa95e008b391f399c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870a8388be904e30b84c8f53ead30f25" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 44291 } } peers { permanent_uuid: "62a20e37a2ea452aa95e008b391f399c" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40873 } } peers { permanent_uuid: "f9dc15d8230743249730a768672a33a3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 44535 } }
I20260926 16:42:09.267311 22603 raft_consensus.cc:374] T d7671ff7b7054faeb3a6208f0d264c50 P 62a20e37a2ea452aa95e008b391f399c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:09.267385 22603 raft_consensus.cc:729] T d7671ff7b7054faeb3a6208f0d264c50 P 62a20e37a2ea452aa95e008b391f399c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62a20e37a2ea452aa95e008b391f399c, State: Initialized, Role: FOLLOWER
I20260926 16:42:09.267812 22603 consensus_queue.cc:260] T d7671ff7b7054faeb3a6208f0d264c50 P 62a20e37a2ea452aa95e008b391f399c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870a8388be904e30b84c8f53ead30f25" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 44291 } } peers { permanent_uuid: "62a20e37a2ea452aa95e008b391f399c" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40873 } } peers { permanent_uuid: "f9dc15d8230743249730a768672a33a3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 44535 } }
I20260926 16:42:09.268663 22448 heartbeater.cc:503] Master 127.21.12.126:41565 was elected leader, sending a full tablet report...
I20260926 16:42:09.268893 22603 ts_tablet_manager.cc:1428] T d7671ff7b7054faeb3a6208f0d264c50 P 62a20e37a2ea452aa95e008b391f399c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260926 16:42:09.433434 22583 tablet.cc:2369] T d7671ff7b7054faeb3a6208f0d264c50 P f9dc15d8230743249730a768672a33a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:42:09.479260 22449 tablet.cc:2369] T d7671ff7b7054faeb3a6208f0d264c50 P 62a20e37a2ea452aa95e008b391f399c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:42:09.617009 22604 raft_consensus.cc:482] T d7671ff7b7054faeb3a6208f0d264c50 P 870a8388be904e30b84c8f53ead30f25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:42:09.617218 22604 raft_consensus.cc:504] T d7671ff7b7054faeb3a6208f0d264c50 P 870a8388be904e30b84c8f53ead30f25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870a8388be904e30b84c8f53ead30f25" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 44291 } } peers { permanent_uuid: "62a20e37a2ea452aa95e008b391f399c" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40873 } } peers { permanent_uuid: "f9dc15d8230743249730a768672a33a3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 44535 } }
I20260926 16:42:09.618355 22604 leader_election.cc:288] T d7671ff7b7054faeb3a6208f0d264c50 P 870a8388be904e30b84c8f53ead30f25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62a20e37a2ea452aa95e008b391f399c (127.21.12.66:40873), f9dc15d8230743249730a768672a33a3 (127.21.12.67:44535)
I20260926 16:42:09.622445 22598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:09.618794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:39335 (local address 127.21.12.67:44535)
0926 16:42:09.618929 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:09.618937 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:09.619116 (+ 179us) server_negotiation.cc:407] Connection header received
0926 16:42:09.619235 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:09.619238 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:09.619299 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:09.619378 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:09.619716 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:09.620247 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:09.621447 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:09.621843 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:09.621973 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:09.622027 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:09.622243 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 16:42:09.622316 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 16:42:09.622460 22609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:09.618678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:40873 (local address 127.21.12.65:43195)
0926 16:42:09.619101 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:09.619116 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:09.619224 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:09.619514 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:09.619517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:09.619531 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:09.619627 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:09.619632 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:09.620392 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:09.620395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:09.621271 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:09.621279 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:09.621994 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:09.621998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:09.622136 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:09.622156 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 16:42:09.622254 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 16:42:09.622317 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":20,"thread_start_us":72,"threads_started":1}
I20260926 16:42:09.622469 22608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:09.618657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.67:44535 (local address 127.21.12.65:39335)
0926 16:42:09.619016 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:09.619028 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:09.619125 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:09.619426 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:09.619432 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:09.619461 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:09.619585 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:09.619603 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:09.620367 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:09.620374 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:09.621299 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:09.621308 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:09.621971 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:09.621975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:09.622113 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:09.622133 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 16:42:09.622194 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 16:42:09.622278 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":80,"thread_start_us":138,"threads_started":1}
I20260926 16:42:09.622475 22597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:09.618755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:43195 (local address 127.21.12.66:40873)
0926 16:42:09.618954 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:09.618959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:09.619213 (+ 254us) server_negotiation.cc:407] Connection header received
0926 16:42:09.619319 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:09.619321 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:09.619393 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:09.619481 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:09.619762 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:09.620254 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:09.621393 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:09.621854 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:09.621980 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:09.622029 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:09.622285 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 16:42:09.622366 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260926 16:42:09.623116 22402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7671ff7b7054faeb3a6208f0d264c50" candidate_uuid: "870a8388be904e30b84c8f53ead30f25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62a20e37a2ea452aa95e008b391f399c" is_pre_election: true
I20260926 16:42:09.623149 22536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7671ff7b7054faeb3a6208f0d264c50" candidate_uuid: "870a8388be904e30b84c8f53ead30f25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9dc15d8230743249730a768672a33a3" is_pre_election: true
I20260926 16:42:09.623412 22402 raft_consensus.cc:2454] T d7671ff7b7054faeb3a6208f0d264c50 P 62a20e37a2ea452aa95e008b391f399c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 870a8388be904e30b84c8f53ead30f25 in term 0.
I20260926 16:42:09.623435 22536 raft_consensus.cc:2454] T d7671ff7b7054faeb3a6208f0d264c50 P f9dc15d8230743249730a768672a33a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 870a8388be904e30b84c8f53ead30f25 in term 0.
I20260926 16:42:09.623893 22204 leader_election.cc:302] T d7671ff7b7054faeb3a6208f0d264c50 P 870a8388be904e30b84c8f53ead30f25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62a20e37a2ea452aa95e008b391f399c, 870a8388be904e30b84c8f53ead30f25; no voters:
I20260926 16:42:09.624186 22604 raft_consensus.cc:2781] T d7671ff7b7054faeb3a6208f0d264c50 P 870a8388be904e30b84c8f53ead30f25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:42:09.624269 22604 raft_consensus.cc:482] T d7671ff7b7054faeb3a6208f0d264c50 P 870a8388be904e30b84c8f53ead30f25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:42:09.624300 22604 raft_consensus.cc:3037] T d7671ff7b7054faeb3a6208f0d264c50 P 870a8388be904e30b84c8f53ead30f25 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:09.625350 22604 raft_consensus.cc:504] T d7671ff7b7054faeb3a6208f0d264c50 P 870a8388be904e30b84c8f53ead30f25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870a8388be904e30b84c8f53ead30f25" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 44291 } } peers { permanent_uuid: "62a20e37a2ea452aa95e008b391f399c" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40873 } } peers { permanent_uuid: "f9dc15d8230743249730a768672a33a3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 44535 } }
I20260926 16:42:09.625720 22604 leader_election.cc:288] T d7671ff7b7054faeb3a6208f0d264c50 P 870a8388be904e30b84c8f53ead30f25 [CANDIDATE]: Term 1 election: Requested vote from peers 62a20e37a2ea452aa95e008b391f399c (127.21.12.66:40873), f9dc15d8230743249730a768672a33a3 (127.21.12.67:44535)
I20260926 16:42:09.626101 22402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7671ff7b7054faeb3a6208f0d264c50" candidate_uuid: "870a8388be904e30b84c8f53ead30f25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62a20e37a2ea452aa95e008b391f399c"
I20260926 16:42:09.626078 22536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7671ff7b7054faeb3a6208f0d264c50" candidate_uuid: "870a8388be904e30b84c8f53ead30f25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9dc15d8230743249730a768672a33a3"
I20260926 16:42:09.626230 22402 raft_consensus.cc:3037] T d7671ff7b7054faeb3a6208f0d264c50 P 62a20e37a2ea452aa95e008b391f399c [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:09.626240 22536 raft_consensus.cc:3037] T d7671ff7b7054faeb3a6208f0d264c50 P f9dc15d8230743249730a768672a33a3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:09.627247 22536 raft_consensus.cc:2454] T d7671ff7b7054faeb3a6208f0d264c50 P f9dc15d8230743249730a768672a33a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 870a8388be904e30b84c8f53ead30f25 in term 1.
I20260926 16:42:09.627255 22402 raft_consensus.cc:2454] T d7671ff7b7054faeb3a6208f0d264c50 P 62a20e37a2ea452aa95e008b391f399c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 870a8388be904e30b84c8f53ead30f25 in term 1.
I20260926 16:42:09.627549 22204 leader_election.cc:302] T d7671ff7b7054faeb3a6208f0d264c50 P 870a8388be904e30b84c8f53ead30f25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62a20e37a2ea452aa95e008b391f399c, 870a8388be904e30b84c8f53ead30f25; no voters:
I20260926 16:42:09.627792 22604 raft_consensus.cc:2781] T d7671ff7b7054faeb3a6208f0d264c50 P 870a8388be904e30b84c8f53ead30f25 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:09.627998 22604 raft_consensus.cc:686] T d7671ff7b7054faeb3a6208f0d264c50 P 870a8388be904e30b84c8f53ead30f25 [term 1 LEADER]: Becoming Leader. State: Replica: 870a8388be904e30b84c8f53ead30f25, State: Running, Role: LEADER
I20260926 16:42:09.628312 22604 consensus_queue.cc:237] T d7671ff7b7054faeb3a6208f0d264c50 P 870a8388be904e30b84c8f53ead30f25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870a8388be904e30b84c8f53ead30f25" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 44291 } } peers { permanent_uuid: "62a20e37a2ea452aa95e008b391f399c" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40873 } } peers { permanent_uuid: "f9dc15d8230743249730a768672a33a3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 44535 } }
I20260926 16:42:09.631304 22123 catalog_manager.cc:5495] T d7671ff7b7054faeb3a6208f0d264c50 P 870a8388be904e30b84c8f53ead30f25 reported cstate change: term changed from 0 to 1, leader changed from <none> to 870a8388be904e30b84c8f53ead30f25 (127.21.12.65). New cstate: current_term: 1 leader_uuid: "870a8388be904e30b84c8f53ead30f25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870a8388be904e30b84c8f53ead30f25" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 44291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "62a20e37a2ea452aa95e008b391f399c" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9dc15d8230743249730a768672a33a3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 44535 } health_report { overall_health: UNKNOWN } } }
Sep 26 16:42:09 dist-test-slave-n1hw krb5kdc[22078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.65@KRBTEST.COM
I20260926 16:42:09.689553 22595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:09.680845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44854 (local address 127.21.12.65:44291)
0926 16:42:09.680997 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:09.681001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:09.681179 (+ 178us) server_negotiation.cc:407] Connection header received
0926 16:42:09.681337 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:09.681340 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:09.681393 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:09.681500 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:09.682269 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:09.682774 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:09.683659 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:09.683876 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:09.683984 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:09.686468 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:09.686479 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:09.686493 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:09.686515 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:09.688044 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:09.688424 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:09.688429 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:09.688430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:09.688520 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:09.688745 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:09.688747 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:09.688748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:09.688981 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:09.689132 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:09.689334 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 16:42:09.689429 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 16:42:09 dist-test-slave-n1hw krb5kdc[22078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440929, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 16:42:09 dist-test-slave-n1hw krb5kdc[22078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440929, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.12.65@KRBTEST.COM
I20260926 16:42:09.715562 22595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:09.706711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44868 (local address 127.21.12.65:44291)
0926 16:42:09.706898 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:09.706903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:09.707048 (+ 145us) server_negotiation.cc:407] Connection header received
0926 16:42:09.707236 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:09.707240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:09.707290 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:09.707368 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:09.708227 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:09.709059 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:09.710125 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:09.710577 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:09.710678 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:09.712895 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:09.712897 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:09.712899 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:09.712920 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:09.714194 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:09.714543 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:09.714545 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:09.714546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:09.714613 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:09.714828 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:09.714830 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:09.714831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:09.714967 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:09.715077 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:09.715280 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 16:42:09.715364 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
Sep 26 16:42:09 dist-test-slave-n1hw krb5kdc[22078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 16:42:09 dist-test-slave-n1hw krb5kdc[22078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.65@KRBTEST.COM
I20260926 16:42:09.738795 22595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:09.731151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44872 (local address 127.21.12.65:44291)
0926 16:42:09.731280 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:09.731283 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:09.731316 (+ 33us) server_negotiation.cc:407] Connection header received
0926 16:42:09.731426 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:09.731428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:09.731464 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:09.731529 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:09.732210 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:09.732677 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:09.733472 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:09.733674 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:09.733745 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:09.736008 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:09.736011 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:09.736013 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:09.736033 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:09.737237 (+ 1204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:09.737590 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:09.737592 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:09.737594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:09.737659 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:09.737986 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:09.737989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:09.737991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:09.738181 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:09.738298 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:09.738493 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 16:42:09.738577 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 16:42:09.740105 22228 tablet_service.cc:2908] Rejecting scan request for tablet d7671ff7b7054faeb3a6208f0d264c50: Uninitialized: clean time has not yet been initialized
I20260926 16:42:09.742210 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 22185
I20260926 16:42:09.748311 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 22319
I20260926 16:42:09.753631 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 22453
I20260926 16:42:09.758683 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 22091
2026-09-26T16:42:09Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2598 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:42:09 dist-test-slave-n1hw krb5kdc[22624](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:42:09 dist-test-slave-n1hw krb5kdc[22624](info): set up 2 sockets
Sep 26 16:42:09 dist-test-slave-n1hw krb5kdc[22624](info): commencing operation
krb5kdc: starting...
W20260926 16:42:10.807503 21553 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Sep 26 16:42:10 dist-test-slave-n1hw krb5kdc[22624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:42:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:42:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
I20260926 16:42:10.930812 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:44995
--webserver_interface=127.21.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:43103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:44995
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:11.037717 22637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:11.038017 22637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:11.038087 22637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:11.041985 22637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:42:11.042065 22637 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 16:42:11.042097 22637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:11.042141 22637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:11.042181 22637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:42:11.042217 22637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:42:11.047356 22637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:43103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:44995
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:44995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:11.048360 22637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:11.049387 22637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:11.057880 22642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:11.057914 22645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:11.057880 22643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:11.058214 22637 server_base.cc:1034] running on GCE node
I20260926 16:42:11.058904 22637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:11.059984 22637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:11.061199 22637 hybrid_clock.cc:648] HybridClock initialized: now 1790440931061183 us; error 27 us; skew 500 ppm
Sep 26 16:42:11 dist-test-slave-n1hw krb5kdc[22624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440931, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:11.067699 22637 init.cc:377] Logged in from keytab as kudu/127.21.12.126@KRBTEST.COM (short username kudu)
I20260926 16:42:11.068689 22637 webserver.cc:460] Webserver started at http://127.21.12.126:34861/ using document root <none> and password file <none>
I20260926 16:42:11.069271 22637 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:11.069365 22637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:11.069617 22637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:11.071776 22637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "40d0c468491848c889c031654666e87d"
format_stamp: "Formatted at 2026-09-26 16:42:11 on dist-test-slave-n1hw"
server_key: "4cf72a2a7f22adfaed0e0d3ce4d2cedb"
server_key_iv: "b9a5700f90b8e019ab57022ced29f7e3"
server_key_version: "encryptionkey@0"
I20260926 16:42:11.072363 22637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "40d0c468491848c889c031654666e87d"
format_stamp: "Formatted at 2026-09-26 16:42:11 on dist-test-slave-n1hw"
server_key: "4cf72a2a7f22adfaed0e0d3ce4d2cedb"
server_key_iv: "b9a5700f90b8e019ab57022ced29f7e3"
server_key_version: "encryptionkey@0"
I20260926 16:42:11.076213 22637 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 16:42:11.078585 22652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:11.079756 22637 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:42:11.079895 22637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "40d0c468491848c889c031654666e87d"
format_stamp: "Formatted at 2026-09-26 16:42:11 on dist-test-slave-n1hw"
server_key: "4cf72a2a7f22adfaed0e0d3ce4d2cedb"
server_key_iv: "b9a5700f90b8e019ab57022ced29f7e3"
server_key_version: "encryptionkey@0"
I20260926 16:42:11.080046 22637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:11.096393 22637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:11.099630 22637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:11.099840 22637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:11.107501 22637 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.126:44995
I20260926 16:42:11.107524 22714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.126:44995 every 8 connection(s)
I20260926 16:42:11.108378 22637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:42:11.110939 22715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:11.116086 22715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40d0c468491848c889c031654666e87d: Bootstrap starting.
I20260926 16:42:11.116479 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 22637
I20260926 16:42:11.116590 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:42:11.116927 21553 external_mini_cluster.cc:1428] Setting key 66dd0000550887d0c7242716cef8e4f1
I20260926 16:42:11.118741 22715 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40d0c468491848c889c031654666e87d: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:11.119431 22715 log.cc:824] T 00000000000000000000000000000000 P 40d0c468491848c889c031654666e87d: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:11.121155 22715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40d0c468491848c889c031654666e87d: No bootstrap required, opened a new log
I20260926 16:42:11.124630 22715 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40d0c468491848c889c031654666e87d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d0c468491848c889c031654666e87d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 44995 } }
I20260926 16:42:11.124918 22715 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40d0c468491848c889c031654666e87d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:11.124975 22715 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40d0c468491848c889c031654666e87d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40d0c468491848c889c031654666e87d, State: Initialized, Role: FOLLOWER
Sep 26 16:42:11 dist-test-slave-n1hw krb5kdc[22624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:42:11.125445 22715 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40d0c468491848c889c031654666e87d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d0c468491848c889c031654666e87d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 44995 } }
I20260926 16:42:11.125599 22715 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40d0c468491848c889c031654666e87d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:42:11.125674 22715 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40d0c468491848c889c031654666e87d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:42:11.125792 22715 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40d0c468491848c889c031654666e87d [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:11.126871 22715 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40d0c468491848c889c031654666e87d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d0c468491848c889c031654666e87d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 44995 } }
I20260926 16:42:11.127244 22715 leader_election.cc:302] T 00000000000000000000000000000000 P 40d0c468491848c889c031654666e87d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40d0c468491848c889c031654666e87d; no voters:
I20260926 16:42:11.127605 22715 leader_election.cc:288] T 00000000000000000000000000000000 P 40d0c468491848c889c031654666e87d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:42:11.127723 22720 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40d0c468491848c889c031654666e87d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:11.127954 22720 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40d0c468491848c889c031654666e87d [term 1 LEADER]: Becoming Leader. State: Replica: 40d0c468491848c889c031654666e87d, State: Running, Role: LEADER
I20260926 16:42:11.128276 22720 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40d0c468491848c889c031654666e87d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d0c468491848c889c031654666e87d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 44995 } }
I20260926 16:42:11.128819 22715 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40d0c468491848c889c031654666e87d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:42:11.130021 22721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40d0c468491848c889c031654666e87d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40d0c468491848c889c031654666e87d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d0c468491848c889c031654666e87d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 44995 } } }
I20260926 16:42:11.130147 22721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40d0c468491848c889c031654666e87d [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:11.130445 22722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40d0c468491848c889c031654666e87d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40d0c468491848c889c031654666e87d. Latest consensus state: current_term: 1 leader_uuid: "40d0c468491848c889c031654666e87d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d0c468491848c889c031654666e87d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 44995 } } }
I20260926 16:42:11.130544 22722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40d0c468491848c889c031654666e87d [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:11.132093 22718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:11.118242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52676 (local address 127.21.12.126:44995)
0926 16:42:11.118641 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:11.118655 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:11.118696 (+ 41us) server_negotiation.cc:407] Connection header received
0926 16:42:11.119337 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:11.119354 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:11.119719 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:11.120033 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:11.120804 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:11.121883 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:11.122721 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:11.123177 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:11.123345 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:11.126011 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:11.126015 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:11.126027 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:11.126051 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:11.128225 (+ 2174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:11.128765 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:11.128771 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:11.128776 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:11.128905 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:11.129317 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:11.129321 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:11.129322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:11.129907 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:11.130742 (+ 835us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:11.130918 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 16:42:11.131207 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":119,"threads_started":1}
W20260926 16:42:11.133571 22735 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 40d0c468491848c889c031654666e87d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 16:42:11.133628 22735 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 16:42:11.133714 22729 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:42:11.134589 22729 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:42:11.139299 22729 catalog_manager.cc:1347] Generated new cluster ID: bcfdf231e2954449b7b1654fe41ea260
I20260926 16:42:11.139358 22729 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:42:11.150295 22729 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:42:11.151495 22729 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:42:11.159204 22729 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40d0c468491848c889c031654666e87d: Generated new TSK 0
I20260926 16:42:11.159863 22729 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab.
Entry for principal kudu/127.21.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab.
I20260926 16:42:11.188422 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.65:0
--local_ip_for_outbound_sockets=127.21.12.65
--webserver_interface=127.21.12.65
--webserver_port=0
--tserver_master_addrs=127.21.12.126:44995
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab
--principal=kudu/127.21.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:43103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:11.301220 22741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:11.301489 22741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:11.301537 22741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:11.305541 22741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:11.305598 22741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:11.305639 22741 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 16:42:11.305720 22741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.65
I20260926 16:42:11.311323 22741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:43103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab
--principal=kudu/127.21.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.12.126:44995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.65
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:11.312326 22741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:11.313304 22741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 16:42:11.322657 22741 server_base.cc:1034] running on GCE node
W20260926 16:42:11.322614 22746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:11.322674 22747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:11.322614 22749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:11.323195 22741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:11.323836 22741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:11.324997 22741 hybrid_clock.cc:648] HybridClock initialized: now 1790440931324976 us; error 30 us; skew 500 ppm
Sep 26 16:42:11 dist-test-slave-n1hw krb5kdc[22624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440931, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:11.327696 22741 init.cc:377] Logged in from keytab as kudu/127.21.12.65@KRBTEST.COM (short username kudu)
I20260926 16:42:11.328482 22741 webserver.cc:460] Webserver started at http://127.21.12.65:34059/ using document root <none> and password file <none>
I20260926 16:42:11.329036 22741 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:11.329129 22741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:11.329365 22741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:11.331377 22741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-0/data/instance:
uuid: "5b1b9e18aaab4efe894b91405689b328"
format_stamp: "Formatted at 2026-09-26 16:42:11 on dist-test-slave-n1hw"
server_key: "2f2f04b36f553048257c49f036cf841e"
server_key_iv: "8fae6d1b4cb41eaa809e0b8e2c71c7ed"
server_key_version: "encryptionkey@0"
I20260926 16:42:11.331933 22741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance:
uuid: "5b1b9e18aaab4efe894b91405689b328"
format_stamp: "Formatted at 2026-09-26 16:42:11 on dist-test-slave-n1hw"
server_key: "2f2f04b36f553048257c49f036cf841e"
server_key_iv: "8fae6d1b4cb41eaa809e0b8e2c71c7ed"
server_key_version: "encryptionkey@0"
I20260926 16:42:11.335614 22741 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260926 16:42:11.338032 22756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:11.339169 22741 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 16:42:11.339298 22741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-0/wal
uuid: "5b1b9e18aaab4efe894b91405689b328"
format_stamp: "Formatted at 2026-09-26 16:42:11 on dist-test-slave-n1hw"
server_key: "2f2f04b36f553048257c49f036cf841e"
server_key_iv: "8fae6d1b4cb41eaa809e0b8e2c71c7ed"
server_key_version: "encryptionkey@0"
I20260926 16:42:11.339406 22741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:11.366729 22741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:11.369657 22741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:11.369877 22741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:11.371259 22741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:11.371305 22741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:11.371368 22741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:11.371399 22741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 16:42:11 dist-test-slave-n1hw krb5kdc[22624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440931, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:42:11.381922 22741 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.65:45435
I20260926 16:42:11.382056 22871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.65:45435 every 8 connection(s)
I20260926 16:42:11.382992 22741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
I20260926 16:42:11.384410 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 22741
I20260926 16:42:11.384541 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance
I20260926 16:42:11.384908 21553 external_mini_cluster.cc:1428] Setting key 05052e99457f1a620f5663da1ce5ae34
I20260926 16:42:11.386147 22718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:11.372142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:46177 (local address 127.21.12.126:44995)
0926 16:42:11.372282 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:11.372286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:11.373308 (+ 1022us) server_negotiation.cc:407] Connection header received
0926 16:42:11.374502 (+ 1194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:11.374507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:11.374569 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:11.374631 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:11.376341 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:11.377231 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:11.378980 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:11.379273 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:11.379369 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:11.382261 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:11.382264 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:11.382268 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:11.382296 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:11.384003 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:11.384509 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:11.384512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:11.384514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:11.384603 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:11.385024 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:11.385029 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:11.385033 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:11.385224 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:11.385336 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:11.385849 (+ 513us) server_negotiation.cc:299] Negotiation successful
0926 16:42:11.385972 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 16:42:11.386891 22765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:11.372685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:44995 (local address 127.21.12.65:46177)
0926 16:42:11.373150 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:11.373206 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:11.374250 (+ 1044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:11.375070 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:11.375079 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:11.375480 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:11.376161 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:11.376190 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:11.377383 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:11.377386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:11.378844 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:11.378855 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:11.379447 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:11.379455 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:11.379676 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:11.380381 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:11.380406 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:11.382079 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:11.384127 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:11.384133 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:11.384148 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:11.384392 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:11.384724 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:11.384728 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:11.384730 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:11.384911 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:11.385333 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:11.385340 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:11.385568 (+ 228us) client_negotiation.cc:769] Sending connection context
0926 16:42:11.385793 (+ 225us) client_negotiation.cc:240] Negotiation successful
0926 16:42:11.386093 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":131,"threads_started":1}
I20260926 16:42:11.388285 22872 heartbeater.cc:347] Connected to a master server at 127.21.12.126:44995
I20260926 16:42:11.388444 22872 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:11.388913 22872 heartbeater.cc:511] Master 127.21.12.126:44995 requested a full tablet report, sending...
I20260926 16:42:11.390456 22668 ts_manager.cc:194] Registered new tserver with Master: 5b1b9e18aaab4efe894b91405689b328 (127.21.12.65:45435)
I20260926 16:42:11.391798 22668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.65@KRBTEST.COM'} at 127.21.12.65:46177
WARNING: no policy specified for kudu/127.21.12.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.66@KRBTEST.COM" created.
I20260926 16:42:11.398080 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:11.390741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:44995 (local address 127.21.12.65:55255)
0926 16:42:11.391029 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:11.391048 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:11.391152 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:11.391405 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:11.391411 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:11.391640 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:11.391798 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:11.391808 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:11.392847 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:11.392850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:11.393470 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:11.393479 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:11.394013 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:11.394016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:11.394126 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:11.394616 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:11.394630 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:11.394936 (+ 306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:11.396578 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:11.396580 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:11.396582 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:11.396766 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:11.397064 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:11.397066 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:11.397067 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:11.397177 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:11.397527 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:11.397529 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:11.397649 (+ 120us) client_negotiation.cc:769] Sending connection context
0926 16:42:11.397806 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 16:42:11.397937 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":2,"thread_start_us":92,"threads_started":1}
I20260926 16:42:11.398195 22718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:11.390542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:55255 (local address 127.21.12.126:44995)
0926 16:42:11.390656 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:11.390660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:11.391136 (+ 476us) server_negotiation.cc:407] Connection header received
0926 16:42:11.391242 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:11.391245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:11.391287 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:11.391367 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:11.392002 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:11.392714 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:11.393589 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:11.393901 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:11.394010 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:11.395076 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:11.395079 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:11.395082 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:11.395105 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:11.396480 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:11.396861 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:11.396865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:11.396866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:11.396968 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:11.397258 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:11.397260 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:11.397261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:11.397415 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:11.397549 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:11.397883 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 16:42:11.398018 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab.
Entry for principal kudu/127.21.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab.
I20260926 16:42:11.413833 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.66:0
--local_ip_for_outbound_sockets=127.21.12.66
--webserver_interface=127.21.12.66
--webserver_port=0
--tserver_master_addrs=127.21.12.126:44995
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab
--principal=kudu/127.21.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:43103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:11.520342 22881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:11.520567 22881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:11.520608 22881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:11.524457 22881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:11.524503 22881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:11.524538 22881 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 16:42:11.524569 22881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.66
I20260926 16:42:11.529675 22881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:43103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab
--principal=kudu/127.21.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.12.126:44995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.66
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:11.530701 22881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:11.531646 22881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:11.540697 22886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:11.540791 22881 server_base.cc:1034] running on GCE node
W20260926 16:42:11.540723 22889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:11.540686 22887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:11.541356 22881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:11.542016 22881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:11.543172 22881 hybrid_clock.cc:648] HybridClock initialized: now 1790440931543138 us; error 43 us; skew 500 ppm
Sep 26 16:42:11 dist-test-slave-n1hw krb5kdc[22624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440931, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:11.545848 22881 init.cc:377] Logged in from keytab as kudu/127.21.12.66@KRBTEST.COM (short username kudu)
I20260926 16:42:11.546738 22881 webserver.cc:460] Webserver started at http://127.21.12.66:35095/ using document root <none> and password file <none>
I20260926 16:42:11.547246 22881 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:11.547303 22881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:11.547456 22881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:11.549377 22881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-1/data/instance:
uuid: "6189b00bcb6d4d3eab725924c7311502"
format_stamp: "Formatted at 2026-09-26 16:42:11 on dist-test-slave-n1hw"
server_key: "91fd4d61f0de289d5d2f6e45be55fc7d"
server_key_iv: "cb13db5e2a3e8f98cf18ebb02fce2337"
server_key_version: "encryptionkey@0"
I20260926 16:42:11.549966 22881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance:
uuid: "6189b00bcb6d4d3eab725924c7311502"
format_stamp: "Formatted at 2026-09-26 16:42:11 on dist-test-slave-n1hw"
server_key: "91fd4d61f0de289d5d2f6e45be55fc7d"
server_key_iv: "cb13db5e2a3e8f98cf18ebb02fce2337"
server_key_version: "encryptionkey@0"
I20260926 16:42:11.553418 22881 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:42:11.555876 22896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:11.557080 22881 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:42:11.557224 22881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-1/wal
uuid: "6189b00bcb6d4d3eab725924c7311502"
format_stamp: "Formatted at 2026-09-26 16:42:11 on dist-test-slave-n1hw"
server_key: "91fd4d61f0de289d5d2f6e45be55fc7d"
server_key_iv: "cb13db5e2a3e8f98cf18ebb02fce2337"
server_key_version: "encryptionkey@0"
I20260926 16:42:11.557335 22881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:11.574301 22881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:11.577230 22881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:11.577433 22881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:11.579031 22881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:11.579094 22881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:11.579145 22881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:11.579175 22881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 16:42:11 dist-test-slave-n1hw krb5kdc[22624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440931, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.66@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:42:11.589862 22881 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.66:43015
I20260926 16:42:11.590538 23011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.66:43015 every 8 connection(s)
I20260926 16:42:11.590814 22881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
I20260926 16:42:11.592590 22718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:11.579767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:58253 (local address 127.21.12.126:44995)
0926 16:42:11.579929 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:11.579934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:11.580783 (+ 849us) server_negotiation.cc:407] Connection header received
0926 16:42:11.581826 (+ 1043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:11.581829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:11.581896 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:11.581961 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:11.583343 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:11.584282 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:11.585594 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:11.585919 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:11.586035 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:11.588846 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:11.588849 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:11.588851 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:11.588874 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:11.590454 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:11.591025 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:11.591028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:11.591029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:11.591100 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:11.591515 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:11.591520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:11.591523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:11.591733 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:11.591841 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:11.592312 (+ 471us) server_negotiation.cc:299] Negotiation successful
0926 16:42:11.592427 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 16:42:11.593317 22905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:11.580195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:44995 (local address 127.21.12.66:58253)
0926 16:42:11.580633 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:11.580681 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:11.581561 (+ 880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:11.582390 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:11.582397 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:11.582697 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:11.583169 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:11.583189 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:11.584466 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:11.584469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:11.585468 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:11.585477 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:11.586081 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:11.586085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:11.586265 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:11.586824 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:11.586843 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:11.588669 (+ 1826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:11.590568 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:11.590573 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:11.590586 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:11.590922 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:11.591215 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:11.591221 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:11.591224 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:11.591391 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:11.591854 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:11.591862 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:11.592084 (+ 222us) client_negotiation.cc:769] Sending connection context
0926 16:42:11.592271 (+ 187us) client_negotiation.cc:240] Negotiation successful
0926 16:42:11.592564 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":126,"threads_started":1}
I20260926 16:42:11.594483 23012 heartbeater.cc:347] Connected to a master server at 127.21.12.126:44995
I20260926 16:42:11.594635 23012 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:11.595191 23012 heartbeater.cc:511] Master 127.21.12.126:44995 requested a full tablet report, sending...
I20260926 16:42:11.596170 22668 ts_manager.cc:194] Registered new tserver with Master: 6189b00bcb6d4d3eab725924c7311502 (127.21.12.66:43015)
I20260926 16:42:11.596966 22668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.66@KRBTEST.COM'} at 127.21.12.66:58253
I20260926 16:42:11.599610 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 22881
I20260926 16:42:11.599735 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance
I20260926 16:42:11.600065 21553 external_mini_cluster.cc:1428] Setting key bbd7674bdaf402b77705446f947fd657
I20260926 16:42:11.604182 23018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:11.596511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:44995 (local address 127.21.12.66:33767)
0926 16:42:11.596797 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:11.596811 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:11.596901 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:11.597172 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:11.597175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:11.597373 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:11.597493 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:11.597503 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:11.598486 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:11.598489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:11.599024 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:11.599032 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:11.599451 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:11.599458 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:11.599559 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:11.600017 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:11.600031 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:11.600410 (+ 379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:11.602676 (+ 2266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:11.602678 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:11.602680 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:11.602860 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:11.603149 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:11.603152 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:11.603153 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:11.603225 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:11.603664 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:11.603680 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:11.603764 (+ 84us) client_negotiation.cc:769] Sending connection context
0926 16:42:11.603867 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 16:42:11.604006 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":85,"threads_started":1}
I20260926 16:42:11.604187 22718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:11.596572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:33767 (local address 127.21.12.126:44995)
0926 16:42:11.596686 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:11.596689 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:11.596875 (+ 186us) server_negotiation.cc:407] Connection header received
0926 16:42:11.596979 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:11.596981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:11.597014 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:11.597115 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:11.597613 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:11.598358 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:11.599127 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:11.599335 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:11.599415 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:11.600553 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:11.600556 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:11.600559 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:11.600583 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:11.602547 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:11.602956 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:11.602962 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:11.602967 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:11.603060 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:11.603348 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:11.603352 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:11.603354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:11.603556 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:11.603663 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:11.603930 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 16:42:11.604034 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
WARNING: no policy specified for kudu/127.21.12.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab.
Entry for principal kudu/127.21.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab.
I20260926 16:42:11.628841 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.67:0
--local_ip_for_outbound_sockets=127.21.12.67
--webserver_interface=127.21.12.67
--webserver_port=0
--tserver_master_addrs=127.21.12.126:44995
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab
--principal=kudu/127.21.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:43103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:11.737111 23021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:11.737334 23021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:11.737375 23021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:11.741225 23021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:11.741273 23021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:11.741310 23021 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 16:42:11.741343 23021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.67
I20260926 16:42:11.746357 23021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:43103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab
--principal=kudu/127.21.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.12.126:44995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.67
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:11.747318 23021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:11.748261 23021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:11.757349 23029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:11.757436 23027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:11.757617 23026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:11.757647 23021 server_base.cc:1034] running on GCE node
I20260926 16:42:11.758036 23021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:11.758692 23021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:11.759886 23021 hybrid_clock.cc:648] HybridClock initialized: now 1790440931759859 us; error 46 us; skew 500 ppm
Sep 26 16:42:11 dist-test-slave-n1hw krb5kdc[22624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440931, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:11.762696 23021 init.cc:377] Logged in from keytab as kudu/127.21.12.67@KRBTEST.COM (short username kudu)
I20260926 16:42:11.763485 23021 webserver.cc:460] Webserver started at http://127.21.12.67:44121/ using document root <none> and password file <none>
I20260926 16:42:11.764014 23021 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:11.764070 23021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:11.764230 23021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:11.766203 23021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-2/data/instance:
uuid: "ad9da2261498484bafdb6a25a4b218c3"
format_stamp: "Formatted at 2026-09-26 16:42:11 on dist-test-slave-n1hw"
server_key: "eb12c51645789003c08c45f471957900"
server_key_iv: "268d3893784ac7748e71139bd666a240"
server_key_version: "encryptionkey@0"
I20260926 16:42:11.766709 23021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance:
uuid: "ad9da2261498484bafdb6a25a4b218c3"
format_stamp: "Formatted at 2026-09-26 16:42:11 on dist-test-slave-n1hw"
server_key: "eb12c51645789003c08c45f471957900"
server_key_iv: "268d3893784ac7748e71139bd666a240"
server_key_version: "encryptionkey@0"
I20260926 16:42:11.770315 23021 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 16:42:11.772605 23036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:11.773869 23021 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:42:11.773976 23021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-2/wal
uuid: "ad9da2261498484bafdb6a25a4b218c3"
format_stamp: "Formatted at 2026-09-26 16:42:11 on dist-test-slave-n1hw"
server_key: "eb12c51645789003c08c45f471957900"
server_key_iv: "268d3893784ac7748e71139bd666a240"
server_key_version: "encryptionkey@0"
I20260926 16:42:11.774072 23021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:11.801525 23021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:11.804594 23021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:11.804800 23021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:11.806295 23021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:11.806345 23021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:11.806384 23021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:11.806401 23021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 16:42:11 dist-test-slave-n1hw krb5kdc[22624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440931, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.67@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:42:11.817005 23021 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.67:33265
I20260926 16:42:11.817121 23151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.67:33265 every 8 connection(s)
I20260926 16:42:11.817973 23021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
I20260926 16:42:11.820104 22718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:11.807127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:41877 (local address 127.21.12.126:44995)
0926 16:42:11.807291 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:11.807296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:11.808079 (+ 783us) server_negotiation.cc:407] Connection header received
0926 16:42:11.809259 (+ 1180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:11.809265 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:11.809334 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:11.809445 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:11.810960 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:11.811551 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:11.812772 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:11.813000 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:11.813131 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:11.816325 (+ 3194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:11.816328 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:11.816331 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:11.816354 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:11.818111 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:11.818727 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:11.818731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:11.818733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:11.818827 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:11.819216 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:11.819219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:11.819220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:11.819387 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:11.819463 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:11.819885 (+ 422us) server_negotiation.cc:299] Negotiation successful
0926 16:42:11.819987 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 16:42:11.820925 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:11.807478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:44995 (local address 127.21.12.67:41877)
0926 16:42:11.807922 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:11.807969 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:11.808931 (+ 962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:11.809666 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:11.809675 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:11.810113 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:11.810756 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:11.810776 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:11.811681 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:11.811685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:11.812649 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:11.812658 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:11.813357 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:11.813361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:11.813574 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:11.814245 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:11.814269 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:11.816142 (+ 1873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:11.818228 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:11.818234 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:11.818248 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:11.818594 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:11.818920 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:11.818925 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:11.818929 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:11.819069 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:11.819475 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:11.819480 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:11.819689 (+ 209us) client_negotiation.cc:769] Sending connection context
0926 16:42:11.819838 (+ 149us) client_negotiation.cc:240] Negotiation successful
0926 16:42:11.820069 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":134,"threads_started":1}
I20260926 16:42:11.822132 23152 heartbeater.cc:347] Connected to a master server at 127.21.12.126:44995
I20260926 16:42:11.822266 23152 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:11.822839 23152 heartbeater.cc:511] Master 127.21.12.126:44995 requested a full tablet report, sending...
I20260926 16:42:11.823988 22669 ts_manager.cc:194] Registered new tserver with Master: ad9da2261498484bafdb6a25a4b218c3 (127.21.12.67:33265)
I20260926 16:42:11.824735 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 23021
I20260926 16:42:11.824786 22669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.67@KRBTEST.COM'} at 127.21.12.67:41877
I20260926 16:42:11.824859 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance
I20260926 16:42:11.825196 21553 external_mini_cluster.cc:1428] Setting key c138ef3c6f52ba29eaa66fde5bbf532a
I20260926 16:42:11.827278 21553 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:42:11.833206 23158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:11.824423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:44995 (local address 127.21.12.67:43719)
0926 16:42:11.824693 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:11.824708 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:11.824791 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:11.825795 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:11.825798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:11.826039 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:11.826164 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:11.826173 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:11.827285 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:11.827290 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:11.828169 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:11.828183 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:11.828709 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:11.828715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:11.828858 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:11.829474 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:11.829489 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:11.829879 (+ 390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:11.831697 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:11.831703 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:11.831708 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:11.831979 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:11.832283 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:11.832289 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:11.832294 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:11.832393 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:11.832784 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:11.832787 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:11.832878 (+ 91us) client_negotiation.cc:769] Sending connection context
0926 16:42:11.832964 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 16:42:11.833066 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":111,"threads_started":1}
I20260926 16:42:11.833462 22718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:11.824480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:43719 (local address 127.21.12.126:44995)
0926 16:42:11.825559 (+ 1079us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:11.825566 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:11.825581 (+ 15us) server_negotiation.cc:407] Connection header received
0926 16:42:11.825620 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:11.825623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:11.825661 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:11.825756 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:11.826421 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:11.827152 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:11.828344 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:11.828593 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:11.828668 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:11.830013 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:11.830016 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:11.830019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:11.830042 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:11.831564 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:11.832078 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:11.832115 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:11.832117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:11.832189 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:11.832485 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:11.832487 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:11.832488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:11.832641 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:11.832779 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:11.833048 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 16:42:11.833224 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1005}
Sep 26 16:42:11 dist-test-slave-n1hw krb5kdc[22624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440931, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 16:42:11 dist-test-slave-n1hw krb5kdc[22624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440931, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:42:11.852763 22718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:11.845053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52684 (local address 127.21.12.126:44995)
0926 16:42:11.845174 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:11.845177 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:11.845188 (+ 11us) server_negotiation.cc:407] Connection header received
0926 16:42:11.845219 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:11.845221 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:11.845253 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:11.845321 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:11.846117 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:11.846608 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:11.847482 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:11.847690 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:11.847772 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:11.850232 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:11.850235 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:11.850238 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:11.850258 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:11.851409 (+ 1151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:11.851768 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:11.851771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:11.851772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:11.851841 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:11.852063 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:11.852065 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:11.852067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:11.852282 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:11.852381 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:11.852565 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 16:42:11.852651 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 16:42:11.855859 22669 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52684:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:42:11.859092 22669 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:42:11.873538 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:11.868093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:45435 (local address 127.0.0.1:45530)
0926 16:42:11.868992 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:11.869002 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:11.869112 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:11.869584 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:11.869587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:11.869604 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:11.869680 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:11.869686 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:11.870628 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:11.870634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:11.872177 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:11.872188 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:11.873031 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:11.873034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:11.873133 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:11.873168 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 16:42:11.873286 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 16:42:11.873400 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"mutex_wait_us":39,"thread_start_us":88,"threads_started":1}
I20260926 16:42:11.873549 23174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:11.868266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45530 (local address 127.21.12.65:45435)
0926 16:42:11.868887 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:11.868899 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:11.869087 (+ 188us) server_negotiation.cc:407] Connection header received
0926 16:42:11.869250 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:11.869255 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:11.869447 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:11.869609 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:11.869806 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:11.870516 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:11.872312 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:11.872905 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:11.873030 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:11.873210 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:11.873315 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 16:42:11.873400 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":88,"threads_started":1}
I20260926 16:42:11.874207 23169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:11.867382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.67:33265 (local address 127.0.0.1:38650)
0926 16:42:11.868535 (+ 1153us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:11.868552 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:11.868638 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:11.869197 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:11.869204 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:11.869225 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:11.869351 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:11.869357 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:11.871023 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:11.871028 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:11.872160 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:11.872170 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:11.873565 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:11.873569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:11.873744 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:11.873835 (+ 91us) client_negotiation.cc:769] Sending connection context
0926 16:42:11.873936 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 16:42:11.874023 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"mutex_wait_us":104,"spinlock_wait_cycles":50944,"thread_start_us":99,"threads_started":1}
I20260926 16:42:11.874235 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:11.867621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38650 (local address 127.21.12.67:33265)
0926 16:42:11.868482 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:11.868496 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:11.868714 (+ 218us) server_negotiation.cc:407] Connection header received
0926 16:42:11.868805 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:11.868810 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:11.869033 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:11.869216 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:11.869474 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:11.870868 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:11.872294 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:11.873180 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:11.873299 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:11.873491 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:11.874004 (+ 513us) server_negotiation.cc:299] Negotiation successful
0926 16:42:11.874102 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":746,"thread_start_us":113,"threads_started":1}
I20260926 16:42:11.874554 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:11.867432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38390 (local address 127.21.12.66:43015)
0926 16:42:11.868654 (+ 1222us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:11.868668 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:11.868692 (+ 24us) server_negotiation.cc:407] Connection header received
0926 16:42:11.868796 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:11.868802 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:11.869034 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:11.869208 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:11.869704 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:11.870944 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:11.872444 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:11.873278 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:11.873372 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:11.873511 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:11.874291 (+ 780us) server_negotiation.cc:299] Negotiation successful
0926 16:42:11.874393 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1119,"thread_start_us":106,"threads_started":1}
I20260926 16:42:11.874828 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:11.867329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:43015 (local address 127.0.0.1:38390)
0926 16:42:11.868157 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:11.868213 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:11.868347 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:11.869373 (+ 1026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:11.869376 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:11.869392 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:11.869524 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:11.869532 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:11.871065 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:11.871068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:11.872332 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:11.872340 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:11.873565 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:11.873569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:11.874051 (+ 482us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:11.874076 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 16:42:11.874144 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 16:42:11.874207 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"mutex_wait_us":191,"thread_start_us":86,"threads_started":1}
I20260926 16:42:11.876966 23086 tablet_service.cc:1467] Processing CreateTablet for tablet 961f7bf0a7c048c2b21faf70bb7912c2 (DEFAULT_TABLE table=test-table [id=f03c2a7fceb74720ac442355590400e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:11.876959 22806 tablet_service.cc:1467] Processing CreateTablet for tablet 961f7bf0a7c048c2b21faf70bb7912c2 (DEFAULT_TABLE table=test-table [id=f03c2a7fceb74720ac442355590400e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:11.877336 22946 tablet_service.cc:1467] Processing CreateTablet for tablet 961f7bf0a7c048c2b21faf70bb7912c2 (DEFAULT_TABLE table=test-table [id=f03c2a7fceb74720ac442355590400e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:11.878024 22806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 961f7bf0a7c048c2b21faf70bb7912c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:11.878032 23086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 961f7bf0a7c048c2b21faf70bb7912c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:11.878213 22946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 961f7bf0a7c048c2b21faf70bb7912c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:11.885396 23175 tablet_bootstrap.cc:492] T 961f7bf0a7c048c2b21faf70bb7912c2 P ad9da2261498484bafdb6a25a4b218c3: Bootstrap starting.
I20260926 16:42:11.886634 23177 tablet_bootstrap.cc:492] T 961f7bf0a7c048c2b21faf70bb7912c2 P 5b1b9e18aaab4efe894b91405689b328: Bootstrap starting.
I20260926 16:42:11.887990 23175 tablet_bootstrap.cc:654] T 961f7bf0a7c048c2b21faf70bb7912c2 P ad9da2261498484bafdb6a25a4b218c3: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:11.888378 23177 tablet_bootstrap.cc:654] T 961f7bf0a7c048c2b21faf70bb7912c2 P 5b1b9e18aaab4efe894b91405689b328: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:11.889037 23177 log.cc:824] T 961f7bf0a7c048c2b21faf70bb7912c2 P 5b1b9e18aaab4efe894b91405689b328: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:11.889158 23175 log.cc:824] T 961f7bf0a7c048c2b21faf70bb7912c2 P ad9da2261498484bafdb6a25a4b218c3: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:11.889812 23176 tablet_bootstrap.cc:492] T 961f7bf0a7c048c2b21faf70bb7912c2 P 6189b00bcb6d4d3eab725924c7311502: Bootstrap starting.
I20260926 16:42:11.891443 23176 tablet_bootstrap.cc:654] T 961f7bf0a7c048c2b21faf70bb7912c2 P 6189b00bcb6d4d3eab725924c7311502: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:11.891484 23175 tablet_bootstrap.cc:492] T 961f7bf0a7c048c2b21faf70bb7912c2 P ad9da2261498484bafdb6a25a4b218c3: No bootstrap required, opened a new log
I20260926 16:42:11.891484 23177 tablet_bootstrap.cc:492] T 961f7bf0a7c048c2b21faf70bb7912c2 P 5b1b9e18aaab4efe894b91405689b328: No bootstrap required, opened a new log
I20260926 16:42:11.891680 23177 ts_tablet_manager.cc:1397] T 961f7bf0a7c048c2b21faf70bb7912c2 P 5b1b9e18aaab4efe894b91405689b328: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 16:42:11.891680 23175 ts_tablet_manager.cc:1397] T 961f7bf0a7c048c2b21faf70bb7912c2 P ad9da2261498484bafdb6a25a4b218c3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 16:42:11.892167 23176 log.cc:824] T 961f7bf0a7c048c2b21faf70bb7912c2 P 6189b00bcb6d4d3eab725924c7311502: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:11.893903 23176 tablet_bootstrap.cc:492] T 961f7bf0a7c048c2b21faf70bb7912c2 P 6189b00bcb6d4d3eab725924c7311502: No bootstrap required, opened a new log
I20260926 16:42:11.894088 23176 ts_tablet_manager.cc:1397] T 961f7bf0a7c048c2b21faf70bb7912c2 P 6189b00bcb6d4d3eab725924c7311502: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260926 16:42:11.896023 23177 raft_consensus.cc:348] T 961f7bf0a7c048c2b21faf70bb7912c2 P 5b1b9e18aaab4efe894b91405689b328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6189b00bcb6d4d3eab725924c7311502" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43015 } } peers { permanent_uuid: "ad9da2261498484bafdb6a25a4b218c3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33265 } } peers { permanent_uuid: "5b1b9e18aaab4efe894b91405689b328" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45435 } }
I20260926 16:42:11.896311 23177 raft_consensus.cc:374] T 961f7bf0a7c048c2b21faf70bb7912c2 P 5b1b9e18aaab4efe894b91405689b328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:11.896366 23177 raft_consensus.cc:729] T 961f7bf0a7c048c2b21faf70bb7912c2 P 5b1b9e18aaab4efe894b91405689b328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b1b9e18aaab4efe894b91405689b328, State: Initialized, Role: FOLLOWER
I20260926 16:42:11.896377 23175 raft_consensus.cc:348] T 961f7bf0a7c048c2b21faf70bb7912c2 P ad9da2261498484bafdb6a25a4b218c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6189b00bcb6d4d3eab725924c7311502" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43015 } } peers { permanent_uuid: "ad9da2261498484bafdb6a25a4b218c3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33265 } } peers { permanent_uuid: "5b1b9e18aaab4efe894b91405689b328" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45435 } }
I20260926 16:42:11.896621 23175 raft_consensus.cc:374] T 961f7bf0a7c048c2b21faf70bb7912c2 P ad9da2261498484bafdb6a25a4b218c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:11.896684 23175 raft_consensus.cc:729] T 961f7bf0a7c048c2b21faf70bb7912c2 P ad9da2261498484bafdb6a25a4b218c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad9da2261498484bafdb6a25a4b218c3, State: Initialized, Role: FOLLOWER
I20260926 16:42:11.896979 23177 consensus_queue.cc:260] T 961f7bf0a7c048c2b21faf70bb7912c2 P 5b1b9e18aaab4efe894b91405689b328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6189b00bcb6d4d3eab725924c7311502" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43015 } } peers { permanent_uuid: "ad9da2261498484bafdb6a25a4b218c3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33265 } } peers { permanent_uuid: "5b1b9e18aaab4efe894b91405689b328" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45435 } }
I20260926 16:42:11.897078 23175 consensus_queue.cc:260] T 961f7bf0a7c048c2b21faf70bb7912c2 P ad9da2261498484bafdb6a25a4b218c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6189b00bcb6d4d3eab725924c7311502" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43015 } } peers { permanent_uuid: "ad9da2261498484bafdb6a25a4b218c3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33265 } } peers { permanent_uuid: "5b1b9e18aaab4efe894b91405689b328" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45435 } }
I20260926 16:42:11.897913 23176 raft_consensus.cc:348] T 961f7bf0a7c048c2b21faf70bb7912c2 P 6189b00bcb6d4d3eab725924c7311502 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6189b00bcb6d4d3eab725924c7311502" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43015 } } peers { permanent_uuid: "ad9da2261498484bafdb6a25a4b218c3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33265 } } peers { permanent_uuid: "5b1b9e18aaab4efe894b91405689b328" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45435 } }
I20260926 16:42:11.898105 23175 ts_tablet_manager.cc:1428] T 961f7bf0a7c048c2b21faf70bb7912c2 P ad9da2261498484bafdb6a25a4b218c3: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 16:42:11.898105 23177 ts_tablet_manager.cc:1428] T 961f7bf0a7c048c2b21faf70bb7912c2 P 5b1b9e18aaab4efe894b91405689b328: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260926 16:42:11.898118 23176 raft_consensus.cc:374] T 961f7bf0a7c048c2b21faf70bb7912c2 P 6189b00bcb6d4d3eab725924c7311502 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:11.898190 23176 raft_consensus.cc:729] T 961f7bf0a7c048c2b21faf70bb7912c2 P 6189b00bcb6d4d3eab725924c7311502 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6189b00bcb6d4d3eab725924c7311502, State: Initialized, Role: FOLLOWER
I20260926 16:42:11.898478 22872 heartbeater.cc:503] Master 127.21.12.126:44995 was elected leader, sending a full tablet report...
I20260926 16:42:11.898478 23152 heartbeater.cc:503] Master 127.21.12.126:44995 was elected leader, sending a full tablet report...
I20260926 16:42:11.898656 23176 consensus_queue.cc:260] T 961f7bf0a7c048c2b21faf70bb7912c2 P 6189b00bcb6d4d3eab725924c7311502 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6189b00bcb6d4d3eab725924c7311502" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43015 } } peers { permanent_uuid: "ad9da2261498484bafdb6a25a4b218c3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33265 } } peers { permanent_uuid: "5b1b9e18aaab4efe894b91405689b328" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45435 } }
I20260926 16:42:11.899266 23012 heartbeater.cc:503] Master 127.21.12.126:44995 was elected leader, sending a full tablet report...
I20260926 16:42:11.899618 23176 ts_tablet_manager.cc:1428] T 961f7bf0a7c048c2b21faf70bb7912c2 P 6189b00bcb6d4d3eab725924c7311502: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260926 16:42:12.069044 23153 tablet.cc:2369] T 961f7bf0a7c048c2b21faf70bb7912c2 P ad9da2261498484bafdb6a25a4b218c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:42:12.092077 23013 tablet.cc:2369] T 961f7bf0a7c048c2b21faf70bb7912c2 P 6189b00bcb6d4d3eab725924c7311502: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:42:12.116931 23182 raft_consensus.cc:482] T 961f7bf0a7c048c2b21faf70bb7912c2 P ad9da2261498484bafdb6a25a4b218c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:42:12.117274 23182 raft_consensus.cc:504] T 961f7bf0a7c048c2b21faf70bb7912c2 P ad9da2261498484bafdb6a25a4b218c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6189b00bcb6d4d3eab725924c7311502" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43015 } } peers { permanent_uuid: "ad9da2261498484bafdb6a25a4b218c3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33265 } } peers { permanent_uuid: "5b1b9e18aaab4efe894b91405689b328" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45435 } }
I20260926 16:42:12.118527 23182 leader_election.cc:288] T 961f7bf0a7c048c2b21faf70bb7912c2 P ad9da2261498484bafdb6a25a4b218c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6189b00bcb6d4d3eab725924c7311502 (127.21.12.66:43015), 5b1b9e18aaab4efe894b91405689b328 (127.21.12.65:45435)
I20260926 16:42:12.122673 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:12.118912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:52699 (local address 127.21.12.66:43015)
0926 16:42:12.119057 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:12.119062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:12.119078 (+ 16us) server_negotiation.cc:407] Connection header received
0926 16:42:12.119252 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:12.119255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:12.119301 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:12.119370 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:12.119868 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.120371 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.121561 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.122074 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.122217 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:12.122280 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:12.122487 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 16:42:12.122561 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 16:42:12.122673 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:12.118834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:43015 (local address 127.21.12.67:52699)
0926 16:42:12.118977 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:12.119001 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:12.119117 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:12.119566 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:12.119572 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:12.119596 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:12.119743 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:12.119750 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.120481 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.120488 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:12.121352 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:12.121361 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.122219 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.122223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:12.122345 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:12.122362 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 16:42:12.122460 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 16:42:12.122536 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"spinlock_wait_cycles":39680}
I20260926 16:42:12.123413 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:12.119070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:45435 (local address 127.21.12.67:49695)
0926 16:42:12.119447 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:12.119461 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:12.119632 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:12.119988 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:12.119990 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:12.120003 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:12.120485 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:12.120492 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.121266 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.121278 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:12.122153 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:12.122162 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.122996 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.123000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:12.123125 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:12.123148 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 16:42:12.123225 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 16:42:12.123285 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":113,"threads_started":1}
I20260926 16:42:12.123346 22966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "961f7bf0a7c048c2b21faf70bb7912c2" candidate_uuid: "ad9da2261498484bafdb6a25a4b218c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6189b00bcb6d4d3eab725924c7311502" is_pre_election: true
I20260926 16:42:12.123420 23174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:12.119292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:49695 (local address 127.21.12.65:45435)
0926 16:42:12.119421 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:12.119425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:12.119538 (+ 113us) server_negotiation.cc:407] Connection header received
0926 16:42:12.119759 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:12.119762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:12.119809 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:12.119859 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:12.120621 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.121136 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.122275 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.122801 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.122925 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:12.122971 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:12.123259 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 16:42:12.123327 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 16:42:12.123674 22966 raft_consensus.cc:2454] T 961f7bf0a7c048c2b21faf70bb7912c2 P 6189b00bcb6d4d3eab725924c7311502 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad9da2261498484bafdb6a25a4b218c3 in term 0.
I20260926 16:42:12.124025 22826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "961f7bf0a7c048c2b21faf70bb7912c2" candidate_uuid: "ad9da2261498484bafdb6a25a4b218c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b1b9e18aaab4efe894b91405689b328" is_pre_election: true
I20260926 16:42:12.124190 23040 leader_election.cc:302] T 961f7bf0a7c048c2b21faf70bb7912c2 P ad9da2261498484bafdb6a25a4b218c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6189b00bcb6d4d3eab725924c7311502, ad9da2261498484bafdb6a25a4b218c3; no voters:
I20260926 16:42:12.124306 22826 raft_consensus.cc:2454] T 961f7bf0a7c048c2b21faf70bb7912c2 P 5b1b9e18aaab4efe894b91405689b328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad9da2261498484bafdb6a25a4b218c3 in term 0.
I20260926 16:42:12.124442 23182 raft_consensus.cc:2781] T 961f7bf0a7c048c2b21faf70bb7912c2 P ad9da2261498484bafdb6a25a4b218c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:42:12.124529 23182 raft_consensus.cc:482] T 961f7bf0a7c048c2b21faf70bb7912c2 P ad9da2261498484bafdb6a25a4b218c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:42:12.124571 23182 raft_consensus.cc:3037] T 961f7bf0a7c048c2b21faf70bb7912c2 P ad9da2261498484bafdb6a25a4b218c3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:12.126188 23182 raft_consensus.cc:504] T 961f7bf0a7c048c2b21faf70bb7912c2 P ad9da2261498484bafdb6a25a4b218c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6189b00bcb6d4d3eab725924c7311502" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43015 } } peers { permanent_uuid: "ad9da2261498484bafdb6a25a4b218c3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33265 } } peers { permanent_uuid: "5b1b9e18aaab4efe894b91405689b328" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45435 } }
I20260926 16:42:12.126539 23182 leader_election.cc:288] T 961f7bf0a7c048c2b21faf70bb7912c2 P ad9da2261498484bafdb6a25a4b218c3 [CANDIDATE]: Term 1 election: Requested vote from peers 6189b00bcb6d4d3eab725924c7311502 (127.21.12.66:43015), 5b1b9e18aaab4efe894b91405689b328 (127.21.12.65:45435)
I20260926 16:42:12.126899 22966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "961f7bf0a7c048c2b21faf70bb7912c2" candidate_uuid: "ad9da2261498484bafdb6a25a4b218c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6189b00bcb6d4d3eab725924c7311502"
I20260926 16:42:12.126968 22826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "961f7bf0a7c048c2b21faf70bb7912c2" candidate_uuid: "ad9da2261498484bafdb6a25a4b218c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b1b9e18aaab4efe894b91405689b328"
I20260926 16:42:12.127045 22966 raft_consensus.cc:3037] T 961f7bf0a7c048c2b21faf70bb7912c2 P 6189b00bcb6d4d3eab725924c7311502 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:12.127140 22826 raft_consensus.cc:3037] T 961f7bf0a7c048c2b21faf70bb7912c2 P 5b1b9e18aaab4efe894b91405689b328 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:12.128042 22966 raft_consensus.cc:2454] T 961f7bf0a7c048c2b21faf70bb7912c2 P 6189b00bcb6d4d3eab725924c7311502 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad9da2261498484bafdb6a25a4b218c3 in term 1.
I20260926 16:42:12.128190 22826 raft_consensus.cc:2454] T 961f7bf0a7c048c2b21faf70bb7912c2 P 5b1b9e18aaab4efe894b91405689b328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad9da2261498484bafdb6a25a4b218c3 in term 1.
I20260926 16:42:12.128366 23040 leader_election.cc:302] T 961f7bf0a7c048c2b21faf70bb7912c2 P ad9da2261498484bafdb6a25a4b218c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6189b00bcb6d4d3eab725924c7311502, ad9da2261498484bafdb6a25a4b218c3; no voters:
I20260926 16:42:12.128554 23182 raft_consensus.cc:2781] T 961f7bf0a7c048c2b21faf70bb7912c2 P ad9da2261498484bafdb6a25a4b218c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:12.128794 23182 raft_consensus.cc:686] T 961f7bf0a7c048c2b21faf70bb7912c2 P ad9da2261498484bafdb6a25a4b218c3 [term 1 LEADER]: Becoming Leader. State: Replica: ad9da2261498484bafdb6a25a4b218c3, State: Running, Role: LEADER
I20260926 16:42:12.129144 23182 consensus_queue.cc:237] T 961f7bf0a7c048c2b21faf70bb7912c2 P ad9da2261498484bafdb6a25a4b218c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6189b00bcb6d4d3eab725924c7311502" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43015 } } peers { permanent_uuid: "ad9da2261498484bafdb6a25a4b218c3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33265 } } peers { permanent_uuid: "5b1b9e18aaab4efe894b91405689b328" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45435 } }
I20260926 16:42:12.132524 22667 catalog_manager.cc:5495] T 961f7bf0a7c048c2b21faf70bb7912c2 P ad9da2261498484bafdb6a25a4b218c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to ad9da2261498484bafdb6a25a4b218c3 (127.21.12.67). New cstate: current_term: 1 leader_uuid: "ad9da2261498484bafdb6a25a4b218c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6189b00bcb6d4d3eab725924c7311502" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad9da2261498484bafdb6a25a4b218c3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b1b9e18aaab4efe894b91405689b328" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45435 } health_report { overall_health: UNKNOWN } } }
W20260926 16:42:12.134238 22873 tablet.cc:2369] T 961f7bf0a7c048c2b21faf70bb7912c2 P 5b1b9e18aaab4efe894b91405689b328: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:42:12.169248 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:12.165690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38658 (local address 127.21.12.67:33265)
0926 16:42:12.165839 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:12.165842 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:12.165902 (+ 60us) server_negotiation.cc:407] Connection header received
0926 16:42:12.166078 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:12.166080 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:12.166142 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:12.166227 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:42:12.166466 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.166969 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.167829 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.168056 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.168175 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:12.168460 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:42:12.168863 (+ 403us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:42:12.168945 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:12.169037 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 16:42:12.169122 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 16:42:12.174947 22966 raft_consensus.cc:1261] T 961f7bf0a7c048c2b21faf70bb7912c2 P 6189b00bcb6d4d3eab725924c7311502 [term 1 FOLLOWER]: Refusing update from remote peer ad9da2261498484bafdb6a25a4b218c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:42:12.174947 22826 raft_consensus.cc:1261] T 961f7bf0a7c048c2b21faf70bb7912c2 P 5b1b9e18aaab4efe894b91405689b328 [term 1 FOLLOWER]: Refusing update from remote peer ad9da2261498484bafdb6a25a4b218c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:42:12.175784 23185 consensus_queue.cc:1035] T 961f7bf0a7c048c2b21faf70bb7912c2 P ad9da2261498484bafdb6a25a4b218c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6189b00bcb6d4d3eab725924c7311502" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:42:12.175985 23182 consensus_queue.cc:1035] T 961f7bf0a7c048c2b21faf70bb7912c2 P ad9da2261498484bafdb6a25a4b218c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b1b9e18aaab4efe894b91405689b328" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:42:12.182528 23188 mvcc.cc:205] Tried to move back new op lower bound from 7333646058181160960 to 7333646058003619840. Current Snapshot: MvccSnapshot[applied={T|T < 7333646058181160960}]
I20260926 16:42:12.183668 23189 mvcc.cc:205] Tried to move back new op lower bound from 7333646058181160960 to 7333646058003619840. Current Snapshot: MvccSnapshot[applied={T|T < 7333646058181160960}]
I20260926 16:42:12.184417 23190 mvcc.cc:205] Tried to move back new op lower bound from 7333646058181160960 to 7333646058003619840. Current Snapshot: MvccSnapshot[applied={T|T < 7333646058181160960}]
I20260926 16:42:12.193853 22669 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:42:12.194022 22669 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:42:12.196494 22669 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 40d0c468491848c889c031654666e87d: Sending DeleteTablet for 3 replicas of tablet 961f7bf0a7c048c2b21faf70bb7912c2
I20260926 16:42:12.197260 23086 tablet_service.cc:1514] Processing DeleteTablet for tablet 961f7bf0a7c048c2b21faf70bb7912c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:12 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:38650
I20260926 16:42:12.197290 22946 tablet_service.cc:1514] Processing DeleteTablet for tablet 961f7bf0a7c048c2b21faf70bb7912c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:12 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:38390
I20260926 16:42:12.197911 23198 tablet_replica.cc:330] T 961f7bf0a7c048c2b21faf70bb7912c2 P 6189b00bcb6d4d3eab725924c7311502: stopping tablet replica
I20260926 16:42:12.197925 22806 tablet_service.cc:1514] Processing DeleteTablet for tablet 961f7bf0a7c048c2b21faf70bb7912c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:12 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:45530
I20260926 16:42:12.197947 22669 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52684:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 16:42:12.198355 23200 tablet_replica.cc:330] T 961f7bf0a7c048c2b21faf70bb7912c2 P 5b1b9e18aaab4efe894b91405689b328: stopping tablet replica
W20260926 16:42:12.198367 22669 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:42:12.198532 23198 raft_consensus.cc:2229] T 961f7bf0a7c048c2b21faf70bb7912c2 P 6189b00bcb6d4d3eab725924c7311502 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:42:12.198640 23200 raft_consensus.cc:2229] T 961f7bf0a7c048c2b21faf70bb7912c2 P 5b1b9e18aaab4efe894b91405689b328 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:42:12.198935 23200 raft_consensus.cc:2258] T 961f7bf0a7c048c2b21faf70bb7912c2 P 5b1b9e18aaab4efe894b91405689b328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:42:12.198935 23198 raft_consensus.cc:2258] T 961f7bf0a7c048c2b21faf70bb7912c2 P 6189b00bcb6d4d3eab725924c7311502 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:42:12.199268 23199 tablet_replica.cc:330] T 961f7bf0a7c048c2b21faf70bb7912c2 P ad9da2261498484bafdb6a25a4b218c3: stopping tablet replica
I20260926 16:42:12.199518 23199 raft_consensus.cc:2229] T 961f7bf0a7c048c2b21faf70bb7912c2 P ad9da2261498484bafdb6a25a4b218c3 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:42:12.199901 23199 raft_consensus.cc:2258] T 961f7bf0a7c048c2b21faf70bb7912c2 P ad9da2261498484bafdb6a25a4b218c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:42:12.200785 23198 ts_tablet_manager.cc:1905] T 961f7bf0a7c048c2b21faf70bb7912c2 P 6189b00bcb6d4d3eab725924c7311502: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:42:12.201206 23200 ts_tablet_manager.cc:1905] T 961f7bf0a7c048c2b21faf70bb7912c2 P 5b1b9e18aaab4efe894b91405689b328: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:42:12.203639 23198 ts_tablet_manager.cc:1918] T 961f7bf0a7c048c2b21faf70bb7912c2 P 6189b00bcb6d4d3eab725924c7311502: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:42:12.203747 23198 log.cc:1196] T 961f7bf0a7c048c2b21faf70bb7912c2 P 6189b00bcb6d4d3eab725924c7311502: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-1/wal/wals/961f7bf0a7c048c2b21faf70bb7912c2
I20260926 16:42:12.203814 23200 ts_tablet_manager.cc:1918] T 961f7bf0a7c048c2b21faf70bb7912c2 P 5b1b9e18aaab4efe894b91405689b328: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:42:12.203892 23200 log.cc:1196] T 961f7bf0a7c048c2b21faf70bb7912c2 P 5b1b9e18aaab4efe894b91405689b328: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-0/wal/wals/961f7bf0a7c048c2b21faf70bb7912c2
I20260926 16:42:12.204077 23198 ts_tablet_manager.cc:1939] T 961f7bf0a7c048c2b21faf70bb7912c2 P 6189b00bcb6d4d3eab725924c7311502: Deleting consensus metadata
I20260926 16:42:12.204175 23200 ts_tablet_manager.cc:1939] T 961f7bf0a7c048c2b21faf70bb7912c2 P 5b1b9e18aaab4efe894b91405689b328: Deleting consensus metadata
I20260926 16:42:12.204226 22806 tablet_service.cc:1467] Processing CreateTablet for tablet 28d2302f3c94486ba5af6f00ab502e4e (DEFAULT_TABLE table=test-table [id=11312411192d49e3875f69bfb414dfcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:12.204483 22806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28d2302f3c94486ba5af6f00ab502e4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:12.205471 23199 ts_tablet_manager.cc:1905] T 961f7bf0a7c048c2b21faf70bb7912c2 P ad9da2261498484bafdb6a25a4b218c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:42:12.205510 23086 tablet_service.cc:1467] Processing CreateTablet for tablet 28d2302f3c94486ba5af6f00ab502e4e (DEFAULT_TABLE table=test-table [id=11312411192d49e3875f69bfb414dfcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:12.205796 23086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28d2302f3c94486ba5af6f00ab502e4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:12.205856 22656 catalog_manager.cc:4841] TS 6189b00bcb6d4d3eab725924c7311502 (127.21.12.66:43015): tablet 961f7bf0a7c048c2b21faf70bb7912c2 (table test-table [id=f03c2a7fceb74720ac442355590400e0]) successfully deleted
I20260926 16:42:12.206946 23177 tablet_bootstrap.cc:492] T 28d2302f3c94486ba5af6f00ab502e4e P 5b1b9e18aaab4efe894b91405689b328: Bootstrap starting.
I20260926 16:42:12.207486 22946 tablet_service.cc:1467] Processing CreateTablet for tablet 28d2302f3c94486ba5af6f00ab502e4e (DEFAULT_TABLE table=test-table [id=11312411192d49e3875f69bfb414dfcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:12.207716 22946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28d2302f3c94486ba5af6f00ab502e4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:12.207893 23177 tablet_bootstrap.cc:654] T 28d2302f3c94486ba5af6f00ab502e4e P 5b1b9e18aaab4efe894b91405689b328: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:12.208173 23199 ts_tablet_manager.cc:1918] T 961f7bf0a7c048c2b21faf70bb7912c2 P ad9da2261498484bafdb6a25a4b218c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:42:12.208299 23199 log.cc:1196] T 961f7bf0a7c048c2b21faf70bb7912c2 P ad9da2261498484bafdb6a25a4b218c3: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-2/wal/wals/961f7bf0a7c048c2b21faf70bb7912c2
I20260926 16:42:12.208654 23199 ts_tablet_manager.cc:1939] T 961f7bf0a7c048c2b21faf70bb7912c2 P ad9da2261498484bafdb6a25a4b218c3: Deleting consensus metadata
I20260926 16:42:12.209734 22653 catalog_manager.cc:4841] TS ad9da2261498484bafdb6a25a4b218c3 (127.21.12.67:33265): tablet 961f7bf0a7c048c2b21faf70bb7912c2 (table test-table [id=f03c2a7fceb74720ac442355590400e0]) successfully deleted
I20260926 16:42:12.210330 23177 tablet_bootstrap.cc:492] T 28d2302f3c94486ba5af6f00ab502e4e P 5b1b9e18aaab4efe894b91405689b328: No bootstrap required, opened a new log
I20260926 16:42:12.210405 23177 ts_tablet_manager.cc:1397] T 28d2302f3c94486ba5af6f00ab502e4e P 5b1b9e18aaab4efe894b91405689b328: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 16:42:12.210471 23176 tablet_bootstrap.cc:492] T 28d2302f3c94486ba5af6f00ab502e4e P 6189b00bcb6d4d3eab725924c7311502: Bootstrap starting.
I20260926 16:42:12.210497 22656 catalog_manager.cc:4841] TS 5b1b9e18aaab4efe894b91405689b328 (127.21.12.65:45435): tablet 961f7bf0a7c048c2b21faf70bb7912c2 (table test-table [id=f03c2a7fceb74720ac442355590400e0]) successfully deleted
I20260926 16:42:12.210803 23177 raft_consensus.cc:348] T 28d2302f3c94486ba5af6f00ab502e4e P 5b1b9e18aaab4efe894b91405689b328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b9e18aaab4efe894b91405689b328" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45435 } } peers { permanent_uuid: "6189b00bcb6d4d3eab725924c7311502" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43015 } } peers { permanent_uuid: "ad9da2261498484bafdb6a25a4b218c3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33265 } }
I20260926 16:42:12.210945 23177 raft_consensus.cc:374] T 28d2302f3c94486ba5af6f00ab502e4e P 5b1b9e18aaab4efe894b91405689b328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:12.210984 23177 raft_consensus.cc:729] T 28d2302f3c94486ba5af6f00ab502e4e P 5b1b9e18aaab4efe894b91405689b328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b1b9e18aaab4efe894b91405689b328, State: Initialized, Role: FOLLOWER
I20260926 16:42:12.211102 23177 consensus_queue.cc:260] T 28d2302f3c94486ba5af6f00ab502e4e P 5b1b9e18aaab4efe894b91405689b328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b9e18aaab4efe894b91405689b328" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45435 } } peers { permanent_uuid: "6189b00bcb6d4d3eab725924c7311502" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43015 } } peers { permanent_uuid: "ad9da2261498484bafdb6a25a4b218c3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33265 } }
I20260926 16:42:12.211410 23176 tablet_bootstrap.cc:654] T 28d2302f3c94486ba5af6f00ab502e4e P 6189b00bcb6d4d3eab725924c7311502: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:12.211400 23177 ts_tablet_manager.cc:1428] T 28d2302f3c94486ba5af6f00ab502e4e P 5b1b9e18aaab4efe894b91405689b328: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 16:42:12.211956 23175 tablet_bootstrap.cc:492] T 28d2302f3c94486ba5af6f00ab502e4e P ad9da2261498484bafdb6a25a4b218c3: Bootstrap starting.
I20260926 16:42:12.212868 23175 tablet_bootstrap.cc:654] T 28d2302f3c94486ba5af6f00ab502e4e P ad9da2261498484bafdb6a25a4b218c3: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:12.213009 23176 tablet_bootstrap.cc:492] T 28d2302f3c94486ba5af6f00ab502e4e P 6189b00bcb6d4d3eab725924c7311502: No bootstrap required, opened a new log
I20260926 16:42:12.213085 23176 ts_tablet_manager.cc:1397] T 28d2302f3c94486ba5af6f00ab502e4e P 6189b00bcb6d4d3eab725924c7311502: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 16:42:12.213570 23176 raft_consensus.cc:348] T 28d2302f3c94486ba5af6f00ab502e4e P 6189b00bcb6d4d3eab725924c7311502 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b9e18aaab4efe894b91405689b328" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45435 } } peers { permanent_uuid: "6189b00bcb6d4d3eab725924c7311502" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43015 } } peers { permanent_uuid: "ad9da2261498484bafdb6a25a4b218c3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33265 } }
I20260926 16:42:12.213701 23176 raft_consensus.cc:374] T 28d2302f3c94486ba5af6f00ab502e4e P 6189b00bcb6d4d3eab725924c7311502 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:12.213744 23176 raft_consensus.cc:729] T 28d2302f3c94486ba5af6f00ab502e4e P 6189b00bcb6d4d3eab725924c7311502 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6189b00bcb6d4d3eab725924c7311502, State: Initialized, Role: FOLLOWER
I20260926 16:42:12.213896 23176 consensus_queue.cc:260] T 28d2302f3c94486ba5af6f00ab502e4e P 6189b00bcb6d4d3eab725924c7311502 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b9e18aaab4efe894b91405689b328" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45435 } } peers { permanent_uuid: "6189b00bcb6d4d3eab725924c7311502" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43015 } } peers { permanent_uuid: "ad9da2261498484bafdb6a25a4b218c3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33265 } }
I20260926 16:42:12.214084 23175 tablet_bootstrap.cc:492] T 28d2302f3c94486ba5af6f00ab502e4e P ad9da2261498484bafdb6a25a4b218c3: No bootstrap required, opened a new log
I20260926 16:42:12.214162 23175 ts_tablet_manager.cc:1397] T 28d2302f3c94486ba5af6f00ab502e4e P ad9da2261498484bafdb6a25a4b218c3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 16:42:12.214195 23176 ts_tablet_manager.cc:1428] T 28d2302f3c94486ba5af6f00ab502e4e P 6189b00bcb6d4d3eab725924c7311502: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 16:42:12.214618 23175 raft_consensus.cc:348] T 28d2302f3c94486ba5af6f00ab502e4e P ad9da2261498484bafdb6a25a4b218c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b9e18aaab4efe894b91405689b328" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45435 } } peers { permanent_uuid: "6189b00bcb6d4d3eab725924c7311502" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43015 } } peers { permanent_uuid: "ad9da2261498484bafdb6a25a4b218c3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33265 } }
I20260926 16:42:12.214733 23175 raft_consensus.cc:374] T 28d2302f3c94486ba5af6f00ab502e4e P ad9da2261498484bafdb6a25a4b218c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:12.214764 23175 raft_consensus.cc:729] T 28d2302f3c94486ba5af6f00ab502e4e P ad9da2261498484bafdb6a25a4b218c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad9da2261498484bafdb6a25a4b218c3, State: Initialized, Role: FOLLOWER
I20260926 16:42:12.214879 23175 consensus_queue.cc:260] T 28d2302f3c94486ba5af6f00ab502e4e P ad9da2261498484bafdb6a25a4b218c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b9e18aaab4efe894b91405689b328" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45435 } } peers { permanent_uuid: "6189b00bcb6d4d3eab725924c7311502" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43015 } } peers { permanent_uuid: "ad9da2261498484bafdb6a25a4b218c3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33265 } }
I20260926 16:42:12.215096 23175 ts_tablet_manager.cc:1428] T 28d2302f3c94486ba5af6f00ab502e4e P ad9da2261498484bafdb6a25a4b218c3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 16:42:12.279659 23181 raft_consensus.cc:482] T 28d2302f3c94486ba5af6f00ab502e4e P 5b1b9e18aaab4efe894b91405689b328 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:42:12.279825 23181 raft_consensus.cc:504] T 28d2302f3c94486ba5af6f00ab502e4e P 5b1b9e18aaab4efe894b91405689b328 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b9e18aaab4efe894b91405689b328" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45435 } } peers { permanent_uuid: "6189b00bcb6d4d3eab725924c7311502" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43015 } } peers { permanent_uuid: "ad9da2261498484bafdb6a25a4b218c3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33265 } }
I20260926 16:42:12.280917 23181 leader_election.cc:288] T 28d2302f3c94486ba5af6f00ab502e4e P 5b1b9e18aaab4efe894b91405689b328 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6189b00bcb6d4d3eab725924c7311502 (127.21.12.66:43015), ad9da2261498484bafdb6a25a4b218c3 (127.21.12.67:33265)
I20260926 16:42:12.285923 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:12.281246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:42385 (local address 127.21.12.66:43015)
0926 16:42:12.281389 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:12.281393 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:12.281528 (+ 135us) server_negotiation.cc:407] Connection header received
0926 16:42:12.281783 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:12.281786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:12.281843 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:12.281896 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:12.282354 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.282910 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.284701 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.285317 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.285438 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:12.285488 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:12.285688 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 16:42:12.285794 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 16:42:12.285948 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:12.281098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:43015 (local address 127.21.12.65:42385)
0926 16:42:12.281438 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:12.281455 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:12.281607 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:12.282052 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:12.282054 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:12.282066 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:12.282219 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:12.282228 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.283023 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.283029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:12.284558 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:12.284569 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.285460 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.285463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:12.285541 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:12.285562 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 16:42:12.285664 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 16:42:12.285787 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":138,"thread_start_us":103,"threads_started":1}
I20260926 16:42:12.286521 22966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d2302f3c94486ba5af6f00ab502e4e" candidate_uuid: "5b1b9e18aaab4efe894b91405689b328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6189b00bcb6d4d3eab725924c7311502" is_pre_election: true
I20260926 16:42:12.286689 22966 raft_consensus.cc:2454] T 28d2302f3c94486ba5af6f00ab502e4e P 6189b00bcb6d4d3eab725924c7311502 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b1b9e18aaab4efe894b91405689b328 in term 0.
I20260926 16:42:12.286795 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:12.281570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:57799 (local address 127.21.12.67:33265)
0926 16:42:12.281711 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:12.281715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:12.281731 (+ 16us) server_negotiation.cc:407] Connection header received
0926 16:42:12.281806 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:12.281809 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:12.281858 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:12.281941 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:12.282318 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.283097 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.285545 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.286133 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.286248 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:12.286292 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:12.286604 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 16:42:12.286685 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 16:42:12.287102 22760 leader_election.cc:302] T 28d2302f3c94486ba5af6f00ab502e4e P 5b1b9e18aaab4efe894b91405689b328 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b1b9e18aaab4efe894b91405689b328, 6189b00bcb6d4d3eab725924c7311502; no voters:
I20260926 16:42:12.287158 23204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:12.281168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.67:33265 (local address 127.21.12.65:57799)
0926 16:42:12.281576 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:12.281588 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:12.281657 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:12.281973 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:12.281976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:12.281999 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:12.282186 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:12.282196 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.283212 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.283215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:12.284559 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:12.284569 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.286353 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.286357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:12.286462 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:12.286484 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 16:42:12.286944 (+ 460us) client_negotiation.cc:240] Negotiation successful
0926 16:42:12.287024 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":84,"thread_start_us":117,"threads_started":1}
I20260926 16:42:12.287361 23181 raft_consensus.cc:2781] T 28d2302f3c94486ba5af6f00ab502e4e P 5b1b9e18aaab4efe894b91405689b328 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:42:12.287425 23181 raft_consensus.cc:482] T 28d2302f3c94486ba5af6f00ab502e4e P 5b1b9e18aaab4efe894b91405689b328 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:42:12.287456 23181 raft_consensus.cc:3037] T 28d2302f3c94486ba5af6f00ab502e4e P 5b1b9e18aaab4efe894b91405689b328 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:12.287715 23106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d2302f3c94486ba5af6f00ab502e4e" candidate_uuid: "5b1b9e18aaab4efe894b91405689b328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad9da2261498484bafdb6a25a4b218c3" is_pre_election: true
I20260926 16:42:12.287911 23106 raft_consensus.cc:2454] T 28d2302f3c94486ba5af6f00ab502e4e P ad9da2261498484bafdb6a25a4b218c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b1b9e18aaab4efe894b91405689b328 in term 0.
I20260926 16:42:12.288702 23181 raft_consensus.cc:504] T 28d2302f3c94486ba5af6f00ab502e4e P 5b1b9e18aaab4efe894b91405689b328 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b9e18aaab4efe894b91405689b328" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45435 } } peers { permanent_uuid: "6189b00bcb6d4d3eab725924c7311502" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43015 } } peers { permanent_uuid: "ad9da2261498484bafdb6a25a4b218c3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33265 } }
I20260926 16:42:12.289049 23181 leader_election.cc:288] T 28d2302f3c94486ba5af6f00ab502e4e P 5b1b9e18aaab4efe894b91405689b328 [CANDIDATE]: Term 1 election: Requested vote from peers 6189b00bcb6d4d3eab725924c7311502 (127.21.12.66:43015), ad9da2261498484bafdb6a25a4b218c3 (127.21.12.67:33265)
I20260926 16:42:12.289372 22966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d2302f3c94486ba5af6f00ab502e4e" candidate_uuid: "5b1b9e18aaab4efe894b91405689b328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6189b00bcb6d4d3eab725924c7311502"
I20260926 16:42:12.289430 23106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d2302f3c94486ba5af6f00ab502e4e" candidate_uuid: "5b1b9e18aaab4efe894b91405689b328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad9da2261498484bafdb6a25a4b218c3"
I20260926 16:42:12.289479 22966 raft_consensus.cc:3037] T 28d2302f3c94486ba5af6f00ab502e4e P 6189b00bcb6d4d3eab725924c7311502 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:12.289552 23106 raft_consensus.cc:3037] T 28d2302f3c94486ba5af6f00ab502e4e P ad9da2261498484bafdb6a25a4b218c3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:12.290434 22966 raft_consensus.cc:2454] T 28d2302f3c94486ba5af6f00ab502e4e P 6189b00bcb6d4d3eab725924c7311502 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b1b9e18aaab4efe894b91405689b328 in term 1.
I20260926 16:42:12.290649 23106 raft_consensus.cc:2454] T 28d2302f3c94486ba5af6f00ab502e4e P ad9da2261498484bafdb6a25a4b218c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b1b9e18aaab4efe894b91405689b328 in term 1.
I20260926 16:42:12.290732 22760 leader_election.cc:302] T 28d2302f3c94486ba5af6f00ab502e4e P 5b1b9e18aaab4efe894b91405689b328 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b1b9e18aaab4efe894b91405689b328, 6189b00bcb6d4d3eab725924c7311502; no voters:
I20260926 16:42:12.290935 23181 raft_consensus.cc:2781] T 28d2302f3c94486ba5af6f00ab502e4e P 5b1b9e18aaab4efe894b91405689b328 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:12.291198 23181 raft_consensus.cc:686] T 28d2302f3c94486ba5af6f00ab502e4e P 5b1b9e18aaab4efe894b91405689b328 [term 1 LEADER]: Becoming Leader. State: Replica: 5b1b9e18aaab4efe894b91405689b328, State: Running, Role: LEADER
I20260926 16:42:12.291436 23181 consensus_queue.cc:237] T 28d2302f3c94486ba5af6f00ab502e4e P 5b1b9e18aaab4efe894b91405689b328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b9e18aaab4efe894b91405689b328" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45435 } } peers { permanent_uuid: "6189b00bcb6d4d3eab725924c7311502" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43015 } } peers { permanent_uuid: "ad9da2261498484bafdb6a25a4b218c3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33265 } }
I20260926 16:42:12.293627 22669 catalog_manager.cc:5495] T 28d2302f3c94486ba5af6f00ab502e4e P 5b1b9e18aaab4efe894b91405689b328 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b1b9e18aaab4efe894b91405689b328 (127.21.12.65). New cstate: current_term: 1 leader_uuid: "5b1b9e18aaab4efe894b91405689b328" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b9e18aaab4efe894b91405689b328" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6189b00bcb6d4d3eab725924c7311502" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad9da2261498484bafdb6a25a4b218c3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33265 } health_report { overall_health: UNKNOWN } } }
Sep 26 16:42:12 dist-test-slave-n1hw krb5kdc[22624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440931, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:42:12.348821 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:12.339501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:44995 (local address 127.0.0.1:52694)
0926 16:42:12.339771 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:12.339788 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:12.339898 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:12.340169 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:12.340171 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:12.340545 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:12.340637 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:12.340643 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.341668 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.341670 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:12.342290 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:12.342299 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.342876 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.342879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:12.343048 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:12.343695 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:12.343706 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:12.345367 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:12.347332 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:12.347340 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:12.347345 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:12.347564 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:12.347885 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:12.347888 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:12.347889 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:12.347971 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:12.348358 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:12.348363 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:12.348462 (+ 99us) client_negotiation.cc:769] Sending connection context
0926 16:42:12.348552 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 16:42:12.348661 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":95,"threads_started":1}
I20260926 16:42:12.348944 22718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:12.339576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52694 (local address 127.21.12.126:44995)
0926 16:42:12.339743 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:12.339748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:12.339878 (+ 130us) server_negotiation.cc:407] Connection header received
0926 16:42:12.339990 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:12.339993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:12.340045 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:12.340151 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:12.340777 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.341545 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.342394 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.342759 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.342856 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:12.345500 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:12.345504 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:12.345507 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:12.345531 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:12.347215 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:12.347677 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:12.347681 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:12.347683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:12.347774 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:12.348057 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:12.348060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:12.348062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:12.348261 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:12.348354 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:12.348629 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 16:42:12.348754 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 16:42:12.352982 22669 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:52694:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 16:42:12.353700 22669 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:42:12.358618 22806 tablet_service.cc:1467] Processing CreateTablet for tablet c8ba6c2cea5f4c19a7ec7267b79d4cd6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=05c59577c0cd45098c4b5b6ecdc9c0bd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 16:42:12.358660 23086 tablet_service.cc:1467] Processing CreateTablet for tablet c8ba6c2cea5f4c19a7ec7267b79d4cd6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=05c59577c0cd45098c4b5b6ecdc9c0bd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 16:42:12.358677 22946 tablet_service.cc:1467] Processing CreateTablet for tablet c8ba6c2cea5f4c19a7ec7267b79d4cd6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=05c59577c0cd45098c4b5b6ecdc9c0bd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 16:42:12.359001 23086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8ba6c2cea5f4c19a7ec7267b79d4cd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:12.359001 22806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8ba6c2cea5f4c19a7ec7267b79d4cd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:12.359021 22946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8ba6c2cea5f4c19a7ec7267b79d4cd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:12.361795 23177 tablet_bootstrap.cc:492] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 5b1b9e18aaab4efe894b91405689b328: Bootstrap starting.
I20260926 16:42:12.361985 23175 tablet_bootstrap.cc:492] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P ad9da2261498484bafdb6a25a4b218c3: Bootstrap starting.
I20260926 16:42:12.362287 23176 tablet_bootstrap.cc:492] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 6189b00bcb6d4d3eab725924c7311502: Bootstrap starting.
I20260926 16:42:12.362766 23177 tablet_bootstrap.cc:654] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 5b1b9e18aaab4efe894b91405689b328: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:12.362766 23175 tablet_bootstrap.cc:654] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P ad9da2261498484bafdb6a25a4b218c3: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:12.363180 23176 tablet_bootstrap.cc:654] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 6189b00bcb6d4d3eab725924c7311502: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:12.364306 23175 tablet_bootstrap.cc:492] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P ad9da2261498484bafdb6a25a4b218c3: No bootstrap required, opened a new log
I20260926 16:42:12.364403 23175 ts_tablet_manager.cc:1397] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P ad9da2261498484bafdb6a25a4b218c3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 16:42:12.364705 23177 tablet_bootstrap.cc:492] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 5b1b9e18aaab4efe894b91405689b328: No bootstrap required, opened a new log
I20260926 16:42:12.364785 23177 ts_tablet_manager.cc:1397] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 5b1b9e18aaab4efe894b91405689b328: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 16:42:12.364768 23175 raft_consensus.cc:348] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P ad9da2261498484bafdb6a25a4b218c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b9e18aaab4efe894b91405689b328" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45435 } } peers { permanent_uuid: "6189b00bcb6d4d3eab725924c7311502" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43015 } } peers { permanent_uuid: "ad9da2261498484bafdb6a25a4b218c3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33265 } }
I20260926 16:42:12.364890 23175 raft_consensus.cc:374] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P ad9da2261498484bafdb6a25a4b218c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:12.364934 23175 raft_consensus.cc:729] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P ad9da2261498484bafdb6a25a4b218c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad9da2261498484bafdb6a25a4b218c3, State: Initialized, Role: FOLLOWER
I20260926 16:42:12.365068 23175 consensus_queue.cc:260] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P ad9da2261498484bafdb6a25a4b218c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b9e18aaab4efe894b91405689b328" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45435 } } peers { permanent_uuid: "6189b00bcb6d4d3eab725924c7311502" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43015 } } peers { permanent_uuid: "ad9da2261498484bafdb6a25a4b218c3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33265 } }
I20260926 16:42:12.365159 23177 raft_consensus.cc:348] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 5b1b9e18aaab4efe894b91405689b328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b9e18aaab4efe894b91405689b328" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45435 } } peers { permanent_uuid: "6189b00bcb6d4d3eab725924c7311502" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43015 } } peers { permanent_uuid: "ad9da2261498484bafdb6a25a4b218c3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33265 } }
I20260926 16:42:12.365293 23177 raft_consensus.cc:374] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 5b1b9e18aaab4efe894b91405689b328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:12.365330 23177 raft_consensus.cc:729] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 5b1b9e18aaab4efe894b91405689b328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b1b9e18aaab4efe894b91405689b328, State: Initialized, Role: FOLLOWER
I20260926 16:42:12.365406 23175 ts_tablet_manager.cc:1428] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P ad9da2261498484bafdb6a25a4b218c3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 16:42:12.365450 23177 consensus_queue.cc:260] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 5b1b9e18aaab4efe894b91405689b328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b9e18aaab4efe894b91405689b328" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45435 } } peers { permanent_uuid: "6189b00bcb6d4d3eab725924c7311502" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43015 } } peers { permanent_uuid: "ad9da2261498484bafdb6a25a4b218c3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33265 } }
I20260926 16:42:12.365689 23177 ts_tablet_manager.cc:1428] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 5b1b9e18aaab4efe894b91405689b328: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 16:42:12.365926 23193 tablet_replica.cc:439] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P ad9da2261498484bafdb6a25a4b218c3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b9e18aaab4efe894b91405689b328" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45435 } } peers { permanent_uuid: "6189b00bcb6d4d3eab725924c7311502" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43015 } } peers { permanent_uuid: "ad9da2261498484bafdb6a25a4b218c3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33265 } } }
I20260926 16:42:12.366055 23193 tablet_replica.cc:442] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P ad9da2261498484bafdb6a25a4b218c3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:42:12.366227 23176 tablet_bootstrap.cc:492] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 6189b00bcb6d4d3eab725924c7311502: No bootstrap required, opened a new log
I20260926 16:42:12.366209 23181 tablet_replica.cc:439] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 5b1b9e18aaab4efe894b91405689b328: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b9e18aaab4efe894b91405689b328" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45435 } } peers { permanent_uuid: "6189b00bcb6d4d3eab725924c7311502" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43015 } } peers { permanent_uuid: "ad9da2261498484bafdb6a25a4b218c3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33265 } } }
I20260926 16:42:12.366304 23176 ts_tablet_manager.cc:1397] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 6189b00bcb6d4d3eab725924c7311502: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260926 16:42:12.366312 23181 tablet_replica.cc:442] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 5b1b9e18aaab4efe894b91405689b328: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:42:12.366756 23176 raft_consensus.cc:348] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 6189b00bcb6d4d3eab725924c7311502 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b9e18aaab4efe894b91405689b328" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45435 } } peers { permanent_uuid: "6189b00bcb6d4d3eab725924c7311502" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43015 } } peers { permanent_uuid: "ad9da2261498484bafdb6a25a4b218c3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33265 } }
I20260926 16:42:12.366856 23176 raft_consensus.cc:374] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 6189b00bcb6d4d3eab725924c7311502 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:12.366894 23176 raft_consensus.cc:729] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 6189b00bcb6d4d3eab725924c7311502 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6189b00bcb6d4d3eab725924c7311502, State: Initialized, Role: FOLLOWER
I20260926 16:42:12.367015 23176 consensus_queue.cc:260] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 6189b00bcb6d4d3eab725924c7311502 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b9e18aaab4efe894b91405689b328" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45435 } } peers { permanent_uuid: "6189b00bcb6d4d3eab725924c7311502" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43015 } } peers { permanent_uuid: "ad9da2261498484bafdb6a25a4b218c3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33265 } }
I20260926 16:42:12.367280 23176 ts_tablet_manager.cc:1428] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 6189b00bcb6d4d3eab725924c7311502: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 16:42:12.367498 23183 tablet_replica.cc:439] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 6189b00bcb6d4d3eab725924c7311502: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b9e18aaab4efe894b91405689b328" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45435 } } peers { permanent_uuid: "6189b00bcb6d4d3eab725924c7311502" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43015 } } peers { permanent_uuid: "ad9da2261498484bafdb6a25a4b218c3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33265 } } }
I20260926 16:42:12.367619 23183 tablet_replica.cc:442] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 6189b00bcb6d4d3eab725924c7311502: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:42:12.374143 23183 raft_consensus.cc:482] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 6189b00bcb6d4d3eab725924c7311502 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:42:12.374264 23183 raft_consensus.cc:504] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 6189b00bcb6d4d3eab725924c7311502 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b9e18aaab4efe894b91405689b328" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45435 } } peers { permanent_uuid: "6189b00bcb6d4d3eab725924c7311502" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43015 } } peers { permanent_uuid: "ad9da2261498484bafdb6a25a4b218c3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33265 } }
I20260926 16:42:12.375299 23183 leader_election.cc:288] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 6189b00bcb6d4d3eab725924c7311502 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b1b9e18aaab4efe894b91405689b328 (127.21.12.65:45435), ad9da2261498484bafdb6a25a4b218c3 (127.21.12.67:33265)
I20260926 16:42:12.379504 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:12.375624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:41161 (local address 127.21.12.67:33265)
0926 16:42:12.375772 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:12.375777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:12.376012 (+ 235us) server_negotiation.cc:407] Connection header received
0926 16:42:12.376132 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:12.376138 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:12.376186 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:12.376262 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:12.376523 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.377070 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.378522 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.378908 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.379034 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:12.379090 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:12.379311 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 16:42:12.379379 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 16:42:12.379514 23174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:12.375563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:41349 (local address 127.21.12.65:45435)
0926 16:42:12.375830 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:12.375835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:12.375850 (+ 15us) server_negotiation.cc:407] Connection header received
0926 16:42:12.375992 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:12.375995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:12.376041 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:12.376120 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:12.376381 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.376948 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.378513 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.378911 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.379035 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:12.379090 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:12.379323 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 16:42:12.379394 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179}
I20260926 16:42:12.379547 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:12.375465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:45435 (local address 127.21.12.66:41349)
0926 16:42:12.375760 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:12.375775 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:12.375876 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:12.376132 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:12.376135 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:12.376152 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:12.376277 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:12.376282 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.377053 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.377058 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:12.378386 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:12.378396 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.379028 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.379032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:12.379203 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:12.379220 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 16:42:12.379309 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 16:42:12.379384 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":94,"thread_start_us":124,"threads_started":1}
I20260926 16:42:12.379555 23218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:12.375509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.67:33265 (local address 127.21.12.66:41161)
0926 16:42:12.375913 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:12.375928 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:12.376051 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:12.376282 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:12.376285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:12.376300 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:12.376419 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:12.376424 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.377180 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.377184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:12.378399 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:12.378416 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.379026 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.379030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:12.379189 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:12.379209 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 16:42:12.379297 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 16:42:12.379366 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":99,"thread_start_us":129,"threads_started":1}
I20260926 16:42:12.380190 22826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8ba6c2cea5f4c19a7ec7267b79d4cd6" candidate_uuid: "6189b00bcb6d4d3eab725924c7311502" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b1b9e18aaab4efe894b91405689b328" is_pre_election: true
I20260926 16:42:12.380226 23106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8ba6c2cea5f4c19a7ec7267b79d4cd6" candidate_uuid: "6189b00bcb6d4d3eab725924c7311502" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad9da2261498484bafdb6a25a4b218c3" is_pre_election: true
I20260926 16:42:12.380337 22826 raft_consensus.cc:2454] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 5b1b9e18aaab4efe894b91405689b328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6189b00bcb6d4d3eab725924c7311502 in term 0.
I20260926 16:42:12.380352 23106 raft_consensus.cc:2454] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P ad9da2261498484bafdb6a25a4b218c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6189b00bcb6d4d3eab725924c7311502 in term 0.
I20260926 16:42:12.380682 22900 leader_election.cc:302] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 6189b00bcb6d4d3eab725924c7311502 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b1b9e18aaab4efe894b91405689b328, 6189b00bcb6d4d3eab725924c7311502; no voters:
I20260926 16:42:12.380946 23183 raft_consensus.cc:2781] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 6189b00bcb6d4d3eab725924c7311502 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:42:12.381037 23183 raft_consensus.cc:482] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 6189b00bcb6d4d3eab725924c7311502 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:42:12.381089 23183 raft_consensus.cc:3037] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 6189b00bcb6d4d3eab725924c7311502 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:12.382280 23183 raft_consensus.cc:504] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 6189b00bcb6d4d3eab725924c7311502 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b9e18aaab4efe894b91405689b328" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45435 } } peers { permanent_uuid: "6189b00bcb6d4d3eab725924c7311502" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43015 } } peers { permanent_uuid: "ad9da2261498484bafdb6a25a4b218c3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33265 } }
I20260926 16:42:12.382617 23183 leader_election.cc:288] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 6189b00bcb6d4d3eab725924c7311502 [CANDIDATE]: Term 1 election: Requested vote from peers 5b1b9e18aaab4efe894b91405689b328 (127.21.12.65:45435), ad9da2261498484bafdb6a25a4b218c3 (127.21.12.67:33265)
I20260926 16:42:12.383013 22826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8ba6c2cea5f4c19a7ec7267b79d4cd6" candidate_uuid: "6189b00bcb6d4d3eab725924c7311502" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b1b9e18aaab4efe894b91405689b328"
I20260926 16:42:12.383026 23106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8ba6c2cea5f4c19a7ec7267b79d4cd6" candidate_uuid: "6189b00bcb6d4d3eab725924c7311502" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad9da2261498484bafdb6a25a4b218c3"
I20260926 16:42:12.383155 23106 raft_consensus.cc:3037] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P ad9da2261498484bafdb6a25a4b218c3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:12.383152 22826 raft_consensus.cc:3037] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 5b1b9e18aaab4efe894b91405689b328 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:12.384230 23106 raft_consensus.cc:2454] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P ad9da2261498484bafdb6a25a4b218c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6189b00bcb6d4d3eab725924c7311502 in term 1.
I20260926 16:42:12.384240 22826 raft_consensus.cc:2454] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 5b1b9e18aaab4efe894b91405689b328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6189b00bcb6d4d3eab725924c7311502 in term 1.
I20260926 16:42:12.384526 22897 leader_election.cc:302] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 6189b00bcb6d4d3eab725924c7311502 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6189b00bcb6d4d3eab725924c7311502, ad9da2261498484bafdb6a25a4b218c3; no voters:
I20260926 16:42:12.384706 23183 raft_consensus.cc:2781] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 6189b00bcb6d4d3eab725924c7311502 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:12.384924 23183 raft_consensus.cc:686] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 6189b00bcb6d4d3eab725924c7311502 [term 1 LEADER]: Becoming Leader. State: Replica: 6189b00bcb6d4d3eab725924c7311502, State: Running, Role: LEADER
I20260926 16:42:12.385166 23183 consensus_queue.cc:237] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 6189b00bcb6d4d3eab725924c7311502 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b9e18aaab4efe894b91405689b328" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45435 } } peers { permanent_uuid: "6189b00bcb6d4d3eab725924c7311502" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43015 } } peers { permanent_uuid: "ad9da2261498484bafdb6a25a4b218c3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33265 } }
I20260926 16:42:12.386296 23219 tablet_replica.cc:439] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 6189b00bcb6d4d3eab725924c7311502: TxnStatusTablet state changed. Reason: New leader 6189b00bcb6d4d3eab725924c7311502. Latest consensus state: current_term: 1 leader_uuid: "6189b00bcb6d4d3eab725924c7311502" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b9e18aaab4efe894b91405689b328" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45435 } } peers { permanent_uuid: "6189b00bcb6d4d3eab725924c7311502" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43015 } } peers { permanent_uuid: "ad9da2261498484bafdb6a25a4b218c3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33265 } } }
I20260926 16:42:12.386389 23219 tablet_replica.cc:442] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 6189b00bcb6d4d3eab725924c7311502: This TxnStatusTablet replica's current role is: LEADER
I20260926 16:42:12.386646 23221 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 16:42:12.386818 23221 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 16:42:12.389210 22667 catalog_manager.cc:5495] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 6189b00bcb6d4d3eab725924c7311502 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6189b00bcb6d4d3eab725924c7311502 (127.21.12.66). New cstate: current_term: 1 leader_uuid: "6189b00bcb6d4d3eab725924c7311502" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b9e18aaab4efe894b91405689b328" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6189b00bcb6d4d3eab725924c7311502" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad9da2261498484bafdb6a25a4b218c3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33265 } health_report { overall_health: UNKNOWN } } }
I20260926 16:42:12.404078 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:12.400389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38392 (local address 127.21.12.66:43015)
0926 16:42:12.400503 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:12.400508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:12.400647 (+ 139us) server_negotiation.cc:407] Connection header received
0926 16:42:12.400858 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:12.400861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:12.400924 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:12.401018 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:42:12.401269 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.401840 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.402696 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.402941 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.403051 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:12.403335 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:42:12.403709 (+ 374us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:42:12.403830 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:12.403908 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 16:42:12.403955 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 16:42:12.404095 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:12.400338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:43015 (local address 127.0.0.1:38392)
0926 16:42:12.400562 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:12.400575 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:12.400717 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:12.401042 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:12.401045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:12.401053 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 16:42:12.401160 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:12.401172 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.401961 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.401965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:12.402573 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:12.402581 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.403053 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.403056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:12.403144 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:12.403196 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 16:42:12.403812 (+ 616us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 16:42:12.403829 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 16:42:12.403897 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 16:42:12.403981 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260926 16:42:12.406662 22826 raft_consensus.cc:1261] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 5b1b9e18aaab4efe894b91405689b328 [term 1 FOLLOWER]: Refusing update from remote peer 6189b00bcb6d4d3eab725924c7311502: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:42:12.406863 23106 raft_consensus.cc:1261] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P ad9da2261498484bafdb6a25a4b218c3 [term 1 FOLLOWER]: Refusing update from remote peer 6189b00bcb6d4d3eab725924c7311502: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:42:12.407100 23219 consensus_queue.cc:1035] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 6189b00bcb6d4d3eab725924c7311502 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b1b9e18aaab4efe894b91405689b328" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:42:12.407335 23183 consensus_queue.cc:1035] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 6189b00bcb6d4d3eab725924c7311502 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad9da2261498484bafdb6a25a4b218c3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:42:12.408547 23181 tablet_replica.cc:439] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 5b1b9e18aaab4efe894b91405689b328: TxnStatusTablet state changed. Reason: New leader 6189b00bcb6d4d3eab725924c7311502. Latest consensus state: current_term: 1 leader_uuid: "6189b00bcb6d4d3eab725924c7311502" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b9e18aaab4efe894b91405689b328" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45435 } } peers { permanent_uuid: "6189b00bcb6d4d3eab725924c7311502" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43015 } } peers { permanent_uuid: "ad9da2261498484bafdb6a25a4b218c3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33265 } } }
I20260926 16:42:12.408648 23181 tablet_replica.cc:442] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 5b1b9e18aaab4efe894b91405689b328: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:42:12.408854 23193 tablet_replica.cc:439] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P ad9da2261498484bafdb6a25a4b218c3: TxnStatusTablet state changed. Reason: New leader 6189b00bcb6d4d3eab725924c7311502. Latest consensus state: current_term: 1 leader_uuid: "6189b00bcb6d4d3eab725924c7311502" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b9e18aaab4efe894b91405689b328" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45435 } } peers { permanent_uuid: "6189b00bcb6d4d3eab725924c7311502" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43015 } } peers { permanent_uuid: "ad9da2261498484bafdb6a25a4b218c3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33265 } } }
I20260926 16:42:12.408967 23193 tablet_replica.cc:442] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P ad9da2261498484bafdb6a25a4b218c3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:42:12.410491 23183 tablet_replica.cc:439] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 6189b00bcb6d4d3eab725924c7311502: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6189b00bcb6d4d3eab725924c7311502" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b9e18aaab4efe894b91405689b328" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45435 } } peers { permanent_uuid: "6189b00bcb6d4d3eab725924c7311502" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43015 } } peers { permanent_uuid: "ad9da2261498484bafdb6a25a4b218c3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33265 } } }
I20260926 16:42:12.410589 23183 tablet_replica.cc:442] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 6189b00bcb6d4d3eab725924c7311502: This TxnStatusTablet replica's current role is: LEADER
I20260926 16:42:12.411420 23181 tablet_replica.cc:439] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 5b1b9e18aaab4efe894b91405689b328: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6189b00bcb6d4d3eab725924c7311502" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b9e18aaab4efe894b91405689b328" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45435 } } peers { permanent_uuid: "6189b00bcb6d4d3eab725924c7311502" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43015 } } peers { permanent_uuid: "ad9da2261498484bafdb6a25a4b218c3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33265 } } }
I20260926 16:42:12.411514 23181 tablet_replica.cc:442] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 5b1b9e18aaab4efe894b91405689b328: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:42:12.414633 23219 tablet_replica.cc:439] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 6189b00bcb6d4d3eab725924c7311502: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6189b00bcb6d4d3eab725924c7311502" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b9e18aaab4efe894b91405689b328" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45435 } } peers { permanent_uuid: "6189b00bcb6d4d3eab725924c7311502" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43015 } } peers { permanent_uuid: "ad9da2261498484bafdb6a25a4b218c3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33265 } } }
I20260926 16:42:12.414736 23219 tablet_replica.cc:442] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P 6189b00bcb6d4d3eab725924c7311502: This TxnStatusTablet replica's current role is: LEADER
I20260926 16:42:12.415563 23193 tablet_replica.cc:439] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P ad9da2261498484bafdb6a25a4b218c3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6189b00bcb6d4d3eab725924c7311502" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b9e18aaab4efe894b91405689b328" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45435 } } peers { permanent_uuid: "6189b00bcb6d4d3eab725924c7311502" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43015 } } peers { permanent_uuid: "ad9da2261498484bafdb6a25a4b218c3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33265 } } }
I20260926 16:42:12.415683 23193 tablet_replica.cc:442] T c8ba6c2cea5f4c19a7ec7267b79d4cd6 P ad9da2261498484bafdb6a25a4b218c3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:42:12.420025 23174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:12.416456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45544 (local address 127.21.12.65:45435)
0926 16:42:12.416568 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:12.416572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:12.416591 (+ 19us) server_negotiation.cc:407] Connection header received
0926 16:42:12.416649 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:12.416651 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:12.416701 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:12.416814 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:42:12.417047 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.417597 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.418486 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.418834 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.418961 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:12.419236 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:42:12.419642 (+ 406us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:42:12.419770 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:12.419843 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 16:42:12.419896 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 16:42:12.431061 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:12.426891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:57603 (local address 127.21.12.66:43015)
0926 16:42:12.427022 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:12.427026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:12.427201 (+ 175us) server_negotiation.cc:407] Connection header received
0926 16:42:12.427428 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:12.427434 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:12.427514 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:12.427602 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:42:12.428018 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.428908 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.429887 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.430229 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.430335 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:12.430600 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:42:12.430685 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:42:12.430774 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:12.430885 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 16:42:12.430937 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 16:42:12.431070 23229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:12.426816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:43015 (local address 127.21.12.65:57603)
0926 16:42:12.427132 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:12.427146 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:12.427247 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:12.427736 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:12.427741 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:12.427749 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 16:42:12.427884 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:12.427893 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.429026 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.429029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:12.429736 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:12.429745 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.430339 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.430341 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:12.430409 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:12.430449 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 16:42:12.430794 (+ 345us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 16:42:12.430800 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 16:42:12.430866 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 16:42:12.430942 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":127,"threads_started":1}
I20260926 16:42:12.438174 23106 raft_consensus.cc:1261] T 28d2302f3c94486ba5af6f00ab502e4e P ad9da2261498484bafdb6a25a4b218c3 [term 1 FOLLOWER]: Refusing update from remote peer 5b1b9e18aaab4efe894b91405689b328: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:42:12.438347 22966 raft_consensus.cc:1261] T 28d2302f3c94486ba5af6f00ab502e4e P 6189b00bcb6d4d3eab725924c7311502 [term 1 FOLLOWER]: Refusing update from remote peer 5b1b9e18aaab4efe894b91405689b328: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:42:12.438791 23181 consensus_queue.cc:1035] T 28d2302f3c94486ba5af6f00ab502e4e P 5b1b9e18aaab4efe894b91405689b328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad9da2261498484bafdb6a25a4b218c3" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:42:12.439173 23206 consensus_queue.cc:1035] T 28d2302f3c94486ba5af6f00ab502e4e P 5b1b9e18aaab4efe894b91405689b328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6189b00bcb6d4d3eab725924c7311502" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:42:12.461433 23174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:12.457507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:34069 (local address 127.21.12.65:45435)
0926 16:42:12.457617 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:12.457621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:12.457878 (+ 257us) server_negotiation.cc:407] Connection header received
0926 16:42:12.457989 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:12.457991 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:12.458028 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:12.458087 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:42:12.458333 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.458897 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.460200 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.460539 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.460619 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:12.461043 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:42:12.461129 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:42:12.461195 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:12.461299 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 16:42:12.461338 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 16:42:12.461532 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:12.457465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:45435 (local address 127.21.12.66:34069)
0926 16:42:12.457798 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:12.457814 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:12.457914 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:12.458121 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:12.458124 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:12.458132 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 16:42:12.458246 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:12.458251 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.459001 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.459004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:12.460068 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:12.460079 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.460740 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.460743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:12.460836 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:12.460882 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 16:42:12.461204 (+ 322us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 16:42:12.461210 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 16:42:12.461292 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 16:42:12.461369 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":101,"threads_started":1}
I20260926 16:42:12.489027 22669 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:42:12.489210 22669 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:42:12.491472 22669 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 40d0c468491848c889c031654666e87d: Sending DeleteTablet for 3 replicas of tablet 28d2302f3c94486ba5af6f00ab502e4e
I20260926 16:42:12.492129 23086 tablet_service.cc:1514] Processing DeleteTablet for tablet 28d2302f3c94486ba5af6f00ab502e4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:12 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:38650
I20260926 16:42:12.492129 22806 tablet_service.cc:1514] Processing DeleteTablet for tablet 28d2302f3c94486ba5af6f00ab502e4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:12 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:45530
I20260926 16:42:12.492313 22946 tablet_service.cc:1514] Processing DeleteTablet for tablet 28d2302f3c94486ba5af6f00ab502e4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:12 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:38390
I20260926 16:42:12.492476 23200 tablet_replica.cc:330] T 28d2302f3c94486ba5af6f00ab502e4e P 5b1b9e18aaab4efe894b91405689b328: stopping tablet replica
I20260926 16:42:12.492502 23198 tablet_replica.cc:330] T 28d2302f3c94486ba5af6f00ab502e4e P 6189b00bcb6d4d3eab725924c7311502: stopping tablet replica
I20260926 16:42:12.492585 23200 raft_consensus.cc:2229] T 28d2302f3c94486ba5af6f00ab502e4e P 5b1b9e18aaab4efe894b91405689b328 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:42:12.492609 23198 raft_consensus.cc:2229] T 28d2302f3c94486ba5af6f00ab502e4e P 6189b00bcb6d4d3eab725924c7311502 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:42:12.492717 23198 raft_consensus.cc:2258] T 28d2302f3c94486ba5af6f00ab502e4e P 6189b00bcb6d4d3eab725924c7311502 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:42:12.492836 23200 raft_consensus.cc:2258] T 28d2302f3c94486ba5af6f00ab502e4e P 5b1b9e18aaab4efe894b91405689b328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:42:12.492993 23199 tablet_replica.cc:330] T 28d2302f3c94486ba5af6f00ab502e4e P ad9da2261498484bafdb6a25a4b218c3: stopping tablet replica
I20260926 16:42:12.493098 23199 raft_consensus.cc:2229] T 28d2302f3c94486ba5af6f00ab502e4e P ad9da2261498484bafdb6a25a4b218c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:42:12.493193 23199 raft_consensus.cc:2258] T 28d2302f3c94486ba5af6f00ab502e4e P ad9da2261498484bafdb6a25a4b218c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:42:12.493462 23198 ts_tablet_manager.cc:1905] T 28d2302f3c94486ba5af6f00ab502e4e P 6189b00bcb6d4d3eab725924c7311502: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:42:12.494231 23199 ts_tablet_manager.cc:1905] T 28d2302f3c94486ba5af6f00ab502e4e P ad9da2261498484bafdb6a25a4b218c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:42:12.496385 23198 ts_tablet_manager.cc:1918] T 28d2302f3c94486ba5af6f00ab502e4e P 6189b00bcb6d4d3eab725924c7311502: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:42:12.496454 23198 log.cc:1196] T 28d2302f3c94486ba5af6f00ab502e4e P 6189b00bcb6d4d3eab725924c7311502: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-1/wal/wals/28d2302f3c94486ba5af6f00ab502e4e
I20260926 16:42:12.496708 23198 ts_tablet_manager.cc:1939] T 28d2302f3c94486ba5af6f00ab502e4e P 6189b00bcb6d4d3eab725924c7311502: Deleting consensus metadata
I20260926 16:42:12.496958 23199 ts_tablet_manager.cc:1918] T 28d2302f3c94486ba5af6f00ab502e4e P ad9da2261498484bafdb6a25a4b218c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:42:12.497022 23199 log.cc:1196] T 28d2302f3c94486ba5af6f00ab502e4e P ad9da2261498484bafdb6a25a4b218c3: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-2/wal/wals/28d2302f3c94486ba5af6f00ab502e4e
I20260926 16:42:12.497258 23199 ts_tablet_manager.cc:1939] T 28d2302f3c94486ba5af6f00ab502e4e P ad9da2261498484bafdb6a25a4b218c3: Deleting consensus metadata
I20260926 16:42:12.497363 22656 catalog_manager.cc:4841] TS 6189b00bcb6d4d3eab725924c7311502 (127.21.12.66:43015): tablet 28d2302f3c94486ba5af6f00ab502e4e (table test-table [id=11312411192d49e3875f69bfb414dfcd]) successfully deleted
I20260926 16:42:12.498160 23200 ts_tablet_manager.cc:1905] T 28d2302f3c94486ba5af6f00ab502e4e P 5b1b9e18aaab4efe894b91405689b328: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:42:12.500738 23200 ts_tablet_manager.cc:1918] T 28d2302f3c94486ba5af6f00ab502e4e P 5b1b9e18aaab4efe894b91405689b328: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:42:12.500819 23200 log.cc:1196] T 28d2302f3c94486ba5af6f00ab502e4e P 5b1b9e18aaab4efe894b91405689b328: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440924937351-21553-0/minicluster-data/ts-0/wal/wals/28d2302f3c94486ba5af6f00ab502e4e
I20260926 16:42:12.501060 23200 ts_tablet_manager.cc:1939] T 28d2302f3c94486ba5af6f00ab502e4e P 5b1b9e18aaab4efe894b91405689b328: Deleting consensus metadata
I20260926 16:42:12.501736 22656 catalog_manager.cc:4841] TS 5b1b9e18aaab4efe894b91405689b328 (127.21.12.65:45435): tablet 28d2302f3c94486ba5af6f00ab502e4e (table test-table [id=11312411192d49e3875f69bfb414dfcd]) successfully deleted
I20260926 16:42:12.501919 22653 catalog_manager.cc:4841] TS ad9da2261498484bafdb6a25a4b218c3 (127.21.12.67:33265): tablet 28d2302f3c94486ba5af6f00ab502e4e (table test-table [id=11312411192d49e3875f69bfb414dfcd]) successfully deleted
Sep 26 16:42:12 dist-test-slave-n1hw krb5kdc[22624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440931, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.12.65@KRBTEST.COM
I20260926 16:42:12.515076 23174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:12.503281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45556 (local address 127.21.12.65:45435)
0926 16:42:12.503390 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:12.503394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:12.503672 (+ 278us) server_negotiation.cc:407] Connection header received
0926 16:42:12.503795 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:12.503798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:12.503848 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:12.503931 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:12.504800 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.505575 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.506695 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.507037 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.507138 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:12.510913 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:12.510922 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:12.510937 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:12.510963 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:12.513023 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:12.513541 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:12.513547 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:12.513553 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:12.513655 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:12.513989 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:12.513992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:12.513995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:12.514313 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:12.514534 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:12.514764 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 16:42:12.514903 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260926 16:42:12.515491 22866 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45556
I20260926 16:42:12.526440 22718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:12.517166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52706 (local address 127.21.12.126:44995)
0926 16:42:12.517300 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:12.517304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:12.517724 (+ 420us) server_negotiation.cc:407] Connection header received
0926 16:42:12.517917 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:12.517921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:12.517972 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:12.518055 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:12.518847 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.519656 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.520770 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.521116 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.521231 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:12.522830 (+ 1599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:12.522834 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:12.522837 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:12.522867 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:12.524597 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:12.525048 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:12.525051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:12.525053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:12.525136 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:12.525517 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:12.525520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:12.525522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:12.525726 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:12.525879 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:12.526163 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 16:42:12.526284 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260926 16:42:12.527004 22669 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52706
I20260926 16:42:12.528731 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 22741
I20260926 16:42:12.535646 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 22881
I20260926 16:42:12.542126 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 23021
I20260926 16:42:12.548625 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 22637
2026-09-26T16:42:12Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2789 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:42:12 dist-test-slave-n1hw krb5kdc[23242](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:42:12 dist-test-slave-n1hw krb5kdc[23242](info): set up 2 sockets
Sep 26 16:42:12 dist-test-slave-n1hw krb5kdc[23242](info): commencing operation
krb5kdc: starting...
W20260926 16:42:13.587715 21553 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Sep 26 16:42:13 dist-test-slave-n1hw krb5kdc[23242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:42:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:42:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
I20260926 16:42:13.709017 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:33641
--webserver_interface=127.21.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:40887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:33641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:13.817502 23255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:13.817811 23255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:13.817919 23255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:13.821843 23255 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:42:13.821918 23255 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 16:42:13.821949 23255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:13.821990 23255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:13.822021 23255 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:42:13.822052 23255 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:42:13.827216 23255 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:40887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:33641
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:33641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:13.828233 23255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:13.829242 23255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 16:42:13.837558 23255 server_base.cc:1034] running on GCE node
W20260926 16:42:13.837489 23261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:13.837498 23260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:13.837488 23263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:13.838375 23255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:13.839408 23255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:13.840596 23255 hybrid_clock.cc:648] HybridClock initialized: now 1790440933840581 us; error 26 us; skew 500 ppm
Sep 26 16:42:13 dist-test-slave-n1hw krb5kdc[23242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440933, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:13.843447 23255 init.cc:377] Logged in from keytab as kudu/127.21.12.126@KRBTEST.COM (short username kudu)
I20260926 16:42:13.844321 23255 webserver.cc:460] Webserver started at http://127.21.12.126:39155/ using document root <none> and password file <none>
I20260926 16:42:13.844813 23255 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:13.844870 23255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:13.845024 23255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:13.847019 23255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "549ff0b7deb048668009a29152477376"
format_stamp: "Formatted at 2026-09-26 16:42:13 on dist-test-slave-n1hw"
server_key: "83072396373bcfa2dd45b3200b4a0d84"
server_key_iv: "5b539c0019a1afe086c5ed6c2bcef8b8"
server_key_version: "encryptionkey@0"
I20260926 16:42:13.847525 23255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "549ff0b7deb048668009a29152477376"
format_stamp: "Formatted at 2026-09-26 16:42:13 on dist-test-slave-n1hw"
server_key: "83072396373bcfa2dd45b3200b4a0d84"
server_key_iv: "5b539c0019a1afe086c5ed6c2bcef8b8"
server_key_version: "encryptionkey@0"
I20260926 16:42:13.851084 23255 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260926 16:42:13.853318 23270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:13.854576 23255 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:42:13.854674 23255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "549ff0b7deb048668009a29152477376"
format_stamp: "Formatted at 2026-09-26 16:42:13 on dist-test-slave-n1hw"
server_key: "83072396373bcfa2dd45b3200b4a0d84"
server_key_iv: "5b539c0019a1afe086c5ed6c2bcef8b8"
server_key_version: "encryptionkey@0"
I20260926 16:42:13.854763 23255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:13.866052 23255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:13.869206 23255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:13.869357 23255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:13.876842 23255 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.126:33641
I20260926 16:42:13.876925 23332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.126:33641 every 8 connection(s)
I20260926 16:42:13.877874 23255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:42:13.880510 23333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:13.884558 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 23255
I20260926 16:42:13.884670 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:42:13.884922 21553 external_mini_cluster.cc:1428] Setting key a92d09bc1d11e588f76f990a216027ae
I20260926 16:42:13.886391 23333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 549ff0b7deb048668009a29152477376: Bootstrap starting.
I20260926 16:42:13.889034 23333 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 549ff0b7deb048668009a29152477376: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:13.889922 23333 log.cc:824] T 00000000000000000000000000000000 P 549ff0b7deb048668009a29152477376: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:13.891880 23333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 549ff0b7deb048668009a29152477376: No bootstrap required, opened a new log
Sep 26 16:42:13 dist-test-slave-n1hw krb5kdc[23242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:42:13.895519 23333 raft_consensus.cc:348] T 00000000000000000000000000000000 P 549ff0b7deb048668009a29152477376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "549ff0b7deb048668009a29152477376" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 33641 } }
I20260926 16:42:13.895828 23333 raft_consensus.cc:374] T 00000000000000000000000000000000 P 549ff0b7deb048668009a29152477376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:13.895895 23333 raft_consensus.cc:729] T 00000000000000000000000000000000 P 549ff0b7deb048668009a29152477376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 549ff0b7deb048668009a29152477376, State: Initialized, Role: FOLLOWER
I20260926 16:42:13.896359 23333 consensus_queue.cc:260] T 00000000000000000000000000000000 P 549ff0b7deb048668009a29152477376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "549ff0b7deb048668009a29152477376" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 33641 } }
I20260926 16:42:13.896528 23333 raft_consensus.cc:388] T 00000000000000000000000000000000 P 549ff0b7deb048668009a29152477376 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:42:13.896598 23333 raft_consensus.cc:482] T 00000000000000000000000000000000 P 549ff0b7deb048668009a29152477376 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:42:13.896724 23333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 549ff0b7deb048668009a29152477376 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:13.897828 23333 raft_consensus.cc:504] T 00000000000000000000000000000000 P 549ff0b7deb048668009a29152477376 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "549ff0b7deb048668009a29152477376" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 33641 } }
I20260926 16:42:13.898182 23333 leader_election.cc:302] T 00000000000000000000000000000000 P 549ff0b7deb048668009a29152477376 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 549ff0b7deb048668009a29152477376; no voters:
I20260926 16:42:13.898530 23333 leader_election.cc:288] T 00000000000000000000000000000000 P 549ff0b7deb048668009a29152477376 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:42:13.898592 23338 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 549ff0b7deb048668009a29152477376 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:13.898886 23338 raft_consensus.cc:686] T 00000000000000000000000000000000 P 549ff0b7deb048668009a29152477376 [term 1 LEADER]: Becoming Leader. State: Replica: 549ff0b7deb048668009a29152477376, State: Running, Role: LEADER
I20260926 16:42:13.899235 23338 consensus_queue.cc:237] T 00000000000000000000000000000000 P 549ff0b7deb048668009a29152477376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "549ff0b7deb048668009a29152477376" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 33641 } }
I20260926 16:42:13.899868 23333 sys_catalog.cc:564] T 00000000000000000000000000000000 P 549ff0b7deb048668009a29152477376 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:42:13.899978 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:13.886271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56588 (local address 127.21.12.126:33641)
0926 16:42:13.886683 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:13.886701 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:13.886746 (+ 45us) server_negotiation.cc:407] Connection header received
0926 16:42:13.887569 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:13.887594 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:13.887993 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:13.888337 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:13.889002 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:13.890499 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:13.891364 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:13.892011 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:13.892176 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:13.894579 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:13.894587 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:13.894605 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:13.894635 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:13.896544 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:13.897087 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:13.897093 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:13.897098 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:13.897239 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:13.897593 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:13.897596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:13.897598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:13.898151 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:13.898338 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:13.898722 (+ 384us) server_negotiation.cc:299] Negotiation successful
0926 16:42:13.899158 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":119,"threads_started":1}
I20260926 16:42:13.900794 23339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 549ff0b7deb048668009a29152477376 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "549ff0b7deb048668009a29152477376" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "549ff0b7deb048668009a29152477376" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 33641 } } }
I20260926 16:42:13.901185 23339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 549ff0b7deb048668009a29152477376 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:13.901089 23340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 549ff0b7deb048668009a29152477376 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 549ff0b7deb048668009a29152477376. Latest consensus state: current_term: 1 leader_uuid: "549ff0b7deb048668009a29152477376" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "549ff0b7deb048668009a29152477376" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 33641 } } }
I20260926 16:42:13.901515 23340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 549ff0b7deb048668009a29152477376 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:13.902251 23347 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:42:13.905416 23347 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:42:13.910790 23347 catalog_manager.cc:1347] Generated new cluster ID: 32c1cb57c4ee4ca080d3e0bfd91aa7f6
I20260926 16:42:13.910858 23347 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:42:13.918556 23347 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:42:13.919773 23347 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:42:13.927575 23347 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 549ff0b7deb048668009a29152477376: Generated new TSK 0
I20260926 16:42:13.928316 23347 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab.
Entry for principal kudu/127.21.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab.
I20260926 16:42:13.957605 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.65:0
--local_ip_for_outbound_sockets=127.21.12.65
--webserver_interface=127.21.12.65
--webserver_port=0
--tserver_master_addrs=127.21.12.126:33641
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab
--principal=kudu/127.21.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:40887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:14.065562 23359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:14.065891 23359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:14.065989 23359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:14.069941 23359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:14.070026 23359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:14.070096 23359 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 16:42:14.070143 23359 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 16:42:14.070207 23359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.65
I20260926 16:42:14.075376 23359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:40887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab
--principal=kudu/127.21.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.12.126:33641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.65
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:14.076404 23359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:14.077435 23359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:14.086774 23367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:14.086954 23359 server_base.cc:1034] running on GCE node
W20260926 16:42:14.086982 23364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:14.086796 23365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:14.087414 23359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:14.088034 23359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:14.089218 23359 hybrid_clock.cc:648] HybridClock initialized: now 1790440934089206 us; error 23 us; skew 500 ppm
Sep 26 16:42:14 dist-test-slave-n1hw krb5kdc[23242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440934, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:14.091905 23359 init.cc:377] Logged in from keytab as kudu/127.21.12.65@KRBTEST.COM (short username kudu)
I20260926 16:42:14.092777 23359 webserver.cc:460] Webserver started at http://127.21.12.65:45437/ using document root <none> and password file <none>
I20260926 16:42:14.093338 23359 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:14.093425 23359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:14.093647 23359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:14.095570 23359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-0/data/instance:
uuid: "1a25832cf0b44dcbb487d80d238ed3f4"
format_stamp: "Formatted at 2026-09-26 16:42:14 on dist-test-slave-n1hw"
server_key: "a69f4b76638320f31637810aca15beb7"
server_key_iv: "76a6a4e5d260d39244275969f647a918"
server_key_version: "encryptionkey@0"
I20260926 16:42:14.096146 23359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance:
uuid: "1a25832cf0b44dcbb487d80d238ed3f4"
format_stamp: "Formatted at 2026-09-26 16:42:14 on dist-test-slave-n1hw"
server_key: "a69f4b76638320f31637810aca15beb7"
server_key_iv: "76a6a4e5d260d39244275969f647a918"
server_key_version: "encryptionkey@0"
I20260926 16:42:14.099733 23359 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 16:42:14.102344 23374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:14.103628 23359 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:42:14.103775 23359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-0/wal
uuid: "1a25832cf0b44dcbb487d80d238ed3f4"
format_stamp: "Formatted at 2026-09-26 16:42:14 on dist-test-slave-n1hw"
server_key: "a69f4b76638320f31637810aca15beb7"
server_key_iv: "76a6a4e5d260d39244275969f647a918"
server_key_version: "encryptionkey@0"
I20260926 16:42:14.103888 23359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:14.138365 23359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:14.141727 23359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:14.141947 23359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:14.143394 23359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:14.143471 23359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:14.143543 23359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:14.143594 23359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 16:42:14 dist-test-slave-n1hw krb5kdc[23242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440934, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:42:14.154726 23359 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.65:39905
I20260926 16:42:14.155064 23489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.65:39905 every 8 connection(s)
I20260926 16:42:14.155737 23359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
I20260926 16:42:14.157693 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:14.144162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:56439 (local address 127.21.12.126:33641)
0926 16:42:14.144300 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:14.144303 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:14.146093 (+ 1790us) server_negotiation.cc:407] Connection header received
0926 16:42:14.147332 (+ 1239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:14.147336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:14.147392 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:14.147508 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:14.148836 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.149681 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.150935 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.151328 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.151452 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:14.154145 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:14.154149 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:14.154152 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:14.154179 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:14.155703 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:14.156311 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:14.156318 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:14.156321 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:14.156419 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:14.156765 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:14.156768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:14.156769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:14.156935 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:14.157013 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:14.157457 (+ 444us) server_negotiation.cc:299] Negotiation successful
0926 16:42:14.157576 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 16:42:14.158525 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:14.145259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:33641 (local address 127.21.12.65:56439)
0926 16:42:14.145923 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:14.145983 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:14.147037 (+ 1054us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:14.147719 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:14.147726 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:14.148171 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:14.148663 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:14.148683 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.149828 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.149832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:14.150809 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:14.150819 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.151507 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.151512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:14.151732 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:14.152320 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:14.152336 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:14.153984 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:14.155818 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:14.155823 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:14.155837 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:14.156175 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:14.156508 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:14.156514 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:14.156517 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:14.156666 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:14.157027 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:14.157031 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:14.157245 (+ 214us) client_negotiation.cc:769] Sending connection context
0926 16:42:14.157410 (+ 165us) client_negotiation.cc:240] Negotiation successful
0926 16:42:14.157684 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":126,"threads_started":1}
I20260926 16:42:14.159744 23490 heartbeater.cc:347] Connected to a master server at 127.21.12.126:33641
I20260926 16:42:14.159874 23490 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:14.160429 23490 heartbeater.cc:511] Master 127.21.12.126:33641 requested a full tablet report, sending...
I20260926 16:42:14.161950 23286 ts_manager.cc:194] Registered new tserver with Master: 1a25832cf0b44dcbb487d80d238ed3f4 (127.21.12.65:39905)
I20260926 16:42:14.163139 23286 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.65@KRBTEST.COM'} at 127.21.12.65:56439
I20260926 16:42:14.164157 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 23359
I20260926 16:42:14.164268 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance
I20260926 16:42:14.164542 21553 external_mini_cluster.cc:1428] Setting key 8cb5615c49a90ad93c1dab20e03f949d
I20260926 16:42:14.170452 23496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:14.161901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:33641 (local address 127.21.12.65:43629)
0926 16:42:14.162251 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:14.162267 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:14.162346 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:14.163593 (+ 1247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:14.163597 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:14.163814 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:14.163930 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:14.163939 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.165098 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.165101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:14.165805 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:14.165815 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.166328 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.166330 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:14.166402 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:14.166873 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:14.166893 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:14.167168 (+ 275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:14.168977 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:14.168979 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:14.168981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:14.169161 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:14.169448 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:14.169450 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:14.169451 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:14.169522 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:14.170061 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:14.170064 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:14.170136 (+ 72us) client_negotiation.cc:769] Sending connection context
0926 16:42:14.170231 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 16:42:14.170329 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":83,"threads_started":1}
I20260926 16:42:14.170598 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:14.162019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:43629 (local address 127.21.12.126:33641)
0926 16:42:14.163407 (+ 1388us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:14.163410 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:14.163421 (+ 11us) server_negotiation.cc:407] Connection header received
0926 16:42:14.163449 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:14.163451 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:14.163483 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:14.163545 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:14.164057 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.164790 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.165915 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.166209 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.166320 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:14.167344 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:14.167350 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:14.167355 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:14.167384 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:14.168870 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:14.169258 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:14.169265 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:14.169269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:14.169357 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:14.169612 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:14.169618 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:14.169623 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:14.169918 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:14.170063 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:14.170320 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 16:42:14.170433 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1317}
WARNING: no policy specified for kudu/127.21.12.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab.
Entry for principal kudu/127.21.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab.
I20260926 16:42:14.192708 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.66:0
--local_ip_for_outbound_sockets=127.21.12.66
--webserver_interface=127.21.12.66
--webserver_port=0
--tserver_master_addrs=127.21.12.126:33641
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab
--principal=kudu/127.21.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:40887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:14.300875 23499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:14.301141 23499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:14.301249 23499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:14.305414 23499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:14.305473 23499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:14.305573 23499 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 16:42:14.305598 23499 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 16:42:14.305682 23499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.66
I20260926 16:42:14.312366 23499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:40887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab
--principal=kudu/127.21.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.12.126:33641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.66
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:14.313335 23499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:14.314380 23499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:14.323592 23505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:14.323720 23499 server_base.cc:1034] running on GCE node
W20260926 16:42:14.323621 23507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:14.323638 23504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:14.324239 23499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:14.324849 23499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:14.326030 23499 hybrid_clock.cc:648] HybridClock initialized: now 1790440934326015 us; error 61 us; skew 500 ppm
Sep 26 16:42:14 dist-test-slave-n1hw krb5kdc[23242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440934, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:14.328722 23499 init.cc:377] Logged in from keytab as kudu/127.21.12.66@KRBTEST.COM (short username kudu)
I20260926 16:42:14.329571 23499 webserver.cc:460] Webserver started at http://127.21.12.66:38225/ using document root <none> and password file <none>
I20260926 16:42:14.330166 23499 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:14.330260 23499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:14.330482 23499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:14.332402 23499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-1/data/instance:
uuid: "56c85d4ee801431f8c556a5dea08cf45"
format_stamp: "Formatted at 2026-09-26 16:42:14 on dist-test-slave-n1hw"
server_key: "c959bfe0c5c67413e8a95ab09a4e6a27"
server_key_iv: "c98703ba8b457a7cff2416519c258b03"
server_key_version: "encryptionkey@0"
I20260926 16:42:14.332954 23499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance:
uuid: "56c85d4ee801431f8c556a5dea08cf45"
format_stamp: "Formatted at 2026-09-26 16:42:14 on dist-test-slave-n1hw"
server_key: "c959bfe0c5c67413e8a95ab09a4e6a27"
server_key_iv: "c98703ba8b457a7cff2416519c258b03"
server_key_version: "encryptionkey@0"
I20260926 16:42:14.336460 23499 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260926 16:42:14.338815 23514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:14.340019 23499 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:42:14.340147 23499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-1/wal
uuid: "56c85d4ee801431f8c556a5dea08cf45"
format_stamp: "Formatted at 2026-09-26 16:42:14 on dist-test-slave-n1hw"
server_key: "c959bfe0c5c67413e8a95ab09a4e6a27"
server_key_iv: "c98703ba8b457a7cff2416519c258b03"
server_key_version: "encryptionkey@0"
I20260926 16:42:14.340256 23499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:14.355754 23499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:14.359118 23499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:14.359338 23499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:14.360883 23499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:14.360951 23499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:14.361022 23499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:14.361084 23499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 16:42:14 dist-test-slave-n1hw krb5kdc[23242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440934, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.66@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:42:14.372009 23499 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.66:40961
I20260926 16:42:14.372076 23629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.66:40961 every 8 connection(s)
I20260926 16:42:14.373121 23499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
I20260926 16:42:14.375303 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:14.361705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:51013 (local address 127.21.12.126:33641)
0926 16:42:14.361848 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:14.361852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:14.362904 (+ 1052us) server_negotiation.cc:407] Connection header received
0926 16:42:14.364079 (+ 1175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:14.364086 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:14.364166 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:14.364238 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:14.365720 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.366665 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.368105 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.368465 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.368605 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:14.371518 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:14.371523 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:14.371527 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:14.371556 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:14.373310 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:14.373949 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:14.373953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:14.373955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:14.374042 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:14.374345 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:14.374348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:14.374349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:14.374514 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:14.374590 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:14.375083 (+ 493us) server_negotiation.cc:299] Negotiation successful
0926 16:42:14.375181 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 16:42:14.376127 23523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:14.362237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:33641 (local address 127.21.12.66:51013)
0926 16:42:14.362748 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:14.362792 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:14.363805 (+ 1013us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:14.364586 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:14.364593 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:14.364923 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:14.365558 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:14.365580 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.366813 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.366817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:14.367960 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:14.367974 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.368650 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.368654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:14.368852 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:14.369451 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:14.369467 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:14.371307 (+ 1840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:14.373421 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:14.373426 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:14.373437 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:14.373839 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:14.374131 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:14.374133 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:14.374134 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:14.374266 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:14.374634 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:14.374639 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:14.374858 (+ 219us) client_negotiation.cc:769] Sending connection context
0926 16:42:14.375022 (+ 164us) client_negotiation.cc:240] Negotiation successful
0926 16:42:14.375286 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":135,"threads_started":1}
I20260926 16:42:14.377260 23630 heartbeater.cc:347] Connected to a master server at 127.21.12.126:33641
I20260926 16:42:14.377388 23630 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:14.377939 23630 heartbeater.cc:511] Master 127.21.12.126:33641 requested a full tablet report, sending...
I20260926 16:42:14.378643 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 23499
I20260926 16:42:14.378769 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance
I20260926 16:42:14.379138 21553 external_mini_cluster.cc:1428] Setting key e37395caefec5e39c283709ab064400d
I20260926 16:42:14.379130 23287 ts_manager.cc:194] Registered new tserver with Master: 56c85d4ee801431f8c556a5dea08cf45 (127.21.12.66:40961)
I20260926 16:42:14.379933 23287 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.66@KRBTEST.COM'} at 127.21.12.66:51013
I20260926 16:42:14.387247 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:14.380228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:35363 (local address 127.21.12.126:33641)
0926 16:42:14.380347 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:14.380352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:14.380369 (+ 17us) server_negotiation.cc:407] Connection header received
0926 16:42:14.380409 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:14.380412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:14.380460 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:14.380545 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:14.381076 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.381813 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.382708 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.382943 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.383052 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:14.384085 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:14.384089 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:14.384092 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:14.384114 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:14.385552 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:14.386039 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:14.386042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:14.386044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:14.386124 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:14.386406 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:14.386411 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:14.386412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:14.386615 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:14.386722 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:14.386989 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 16:42:14.387091 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 16:42:14.387259 23636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:14.379500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:33641 (local address 127.21.12.66:35363)
0926 16:42:14.379776 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:14.379793 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:14.379882 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:14.380558 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:14.380561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:14.380772 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:14.380942 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:14.380950 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.381998 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.382001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:14.382586 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:14.382594 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.383054 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.383057 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:14.383147 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:14.383652 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:14.383662 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:14.383952 (+ 290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:14.385663 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:14.385668 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:14.385670 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:14.385939 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:14.386224 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:14.386227 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:14.386229 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:14.386316 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:14.386730 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:14.386735 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:14.386827 (+ 92us) client_negotiation.cc:769] Sending connection context
0926 16:42:14.386936 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 16:42:14.387078 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":18,"thread_start_us":86,"threads_started":1}
WARNING: no policy specified for kudu/127.21.12.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab.
Entry for principal kudu/127.21.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab.
I20260926 16:42:14.409922 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.67:0
--local_ip_for_outbound_sockets=127.21.12.67
--webserver_interface=127.21.12.67
--webserver_port=0
--tserver_master_addrs=127.21.12.126:33641
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab
--principal=kudu/127.21.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:40887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:14.522789 23639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:14.523048 23639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:14.523132 23639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:14.527088 23639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:14.527163 23639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:14.527231 23639 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 16:42:14.527272 23639 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 16:42:14.527339 23639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.67
I20260926 16:42:14.532413 23639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:40887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab
--principal=kudu/127.21.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.12.126:33641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.67
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:14.533377 23639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:14.534389 23639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 16:42:14.543843 23639 server_base.cc:1034] running on GCE node
W20260926 16:42:14.543879 23645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:14.543882 23647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:14.544029 23644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:14.544656 23639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:14.545295 23639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:14.546490 23639 hybrid_clock.cc:648] HybridClock initialized: now 1790440934546482 us; error 29 us; skew 500 ppm
Sep 26 16:42:14 dist-test-slave-n1hw krb5kdc[23242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440934, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:14.549116 23639 init.cc:377] Logged in from keytab as kudu/127.21.12.67@KRBTEST.COM (short username kudu)
I20260926 16:42:14.549996 23639 webserver.cc:460] Webserver started at http://127.21.12.67:38549/ using document root <none> and password file <none>
I20260926 16:42:14.550544 23639 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:14.550632 23639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:14.550879 23639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:14.552953 23639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-2/data/instance:
uuid: "ffedcedd84804834b6a74572c0b94cc8"
format_stamp: "Formatted at 2026-09-26 16:42:14 on dist-test-slave-n1hw"
server_key: "3d7a6412b7e2ba7744221154e4427d69"
server_key_iv: "10054bf5c7ff9caa8c91ba151f915851"
server_key_version: "encryptionkey@0"
I20260926 16:42:14.553514 23639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance:
uuid: "ffedcedd84804834b6a74572c0b94cc8"
format_stamp: "Formatted at 2026-09-26 16:42:14 on dist-test-slave-n1hw"
server_key: "3d7a6412b7e2ba7744221154e4427d69"
server_key_iv: "10054bf5c7ff9caa8c91ba151f915851"
server_key_version: "encryptionkey@0"
I20260926 16:42:14.557111 23639 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 16:42:14.559774 23654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:14.561090 23639 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:42:14.561225 23639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-2/wal
uuid: "ffedcedd84804834b6a74572c0b94cc8"
format_stamp: "Formatted at 2026-09-26 16:42:14 on dist-test-slave-n1hw"
server_key: "3d7a6412b7e2ba7744221154e4427d69"
server_key_iv: "10054bf5c7ff9caa8c91ba151f915851"
server_key_version: "encryptionkey@0"
I20260926 16:42:14.561345 23639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:14.573230 23639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:14.576465 23639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:14.576689 23639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:14.578358 23639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:14.578431 23639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:14.578491 23639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:14.578531 23639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:14.589493 23639 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.67:36983
I20260926 16:42:14.589561 23769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.67:36983 every 8 connection(s)
Sep 26 16:42:14 dist-test-slave-n1hw krb5kdc[23242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440934, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.67@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:42:14.590576 23639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
I20260926 16:42:14.593545 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:14.579037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:54257 (local address 127.21.12.126:33641)
0926 16:42:14.579166 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:14.579170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:14.580878 (+ 1708us) server_negotiation.cc:407] Connection header received
0926 16:42:14.582111 (+ 1233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:14.582114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:14.582173 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:14.582264 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:14.584182 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.585062 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.586885 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.587096 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.587179 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:14.590239 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:14.590242 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:14.590244 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:14.590268 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:14.591804 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:14.592287 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:14.592290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:14.592291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:14.592362 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:14.592665 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:14.592667 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:14.592669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:14.592824 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:14.592903 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:14.593343 (+ 440us) server_negotiation.cc:299] Negotiation successful
0926 16:42:14.593432 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 16:42:14.594332 23664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:14.580160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:33641 (local address 127.21.12.67:54257)
0926 16:42:14.580708 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:14.580758 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:14.581895 (+ 1137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:14.582554 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:14.582567 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:14.583178 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:14.584021 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:14.584045 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.585178 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.585182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:14.586753 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:14.586765 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.587246 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.587254 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:14.587488 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:14.588179 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:14.588205 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:14.590077 (+ 1872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:14.591899 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:14.591903 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:14.591913 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:14.592169 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:14.592441 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:14.592443 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:14.592445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:14.592583 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:14.592910 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:14.592914 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:14.593127 (+ 213us) client_negotiation.cc:769] Sending connection context
0926 16:42:14.593265 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 16:42:14.593482 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":120,"threads_started":1}
I20260926 16:42:14.595453 23770 heartbeater.cc:347] Connected to a master server at 127.21.12.126:33641
I20260926 16:42:14.595608 23770 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:14.595860 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 23639
I20260926 16:42:14.595969 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance
I20260926 16:42:14.596154 23770 heartbeater.cc:511] Master 127.21.12.126:33641 requested a full tablet report, sending...
I20260926 16:42:14.596302 21553 external_mini_cluster.cc:1428] Setting key 17504e389dc8905d6e083b7ece685743
I20260926 16:42:14.597189 23286 ts_manager.cc:194] Registered new tserver with Master: ffedcedd84804834b6a74572c0b94cc8 (127.21.12.67:36983)
I20260926 16:42:14.597973 23286 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.67@KRBTEST.COM'} at 127.21.12.67:54257
I20260926 16:42:14.598592 21553 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:42:14.605371 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:14.598166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:33727 (local address 127.21.12.126:33641)
0926 16:42:14.598256 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:14.598259 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:14.598272 (+ 13us) server_negotiation.cc:407] Connection header received
0926 16:42:14.598328 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:14.598330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:14.598358 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:14.598418 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:14.598961 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.599684 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.600971 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.601219 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.601327 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:14.602384 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:14.602388 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:14.602391 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:14.602412 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:14.603820 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:14.604243 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:14.604248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:14.604250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:14.604325 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:14.604602 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:14.604606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:14.604608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:14.604764 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:14.604869 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:14.605130 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 16:42:14.605231 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 16:42:14.605384 23776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:14.597888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:33641 (local address 127.21.12.67:33727)
0926 16:42:14.598155 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:14.598169 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:14.598256 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:14.598462 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:14.598465 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:14.598671 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:14.598843 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:14.598850 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.599831 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.599837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:14.600856 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:14.600865 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.601329 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.601333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:14.601426 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:14.601950 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:14.601963 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:14.602259 (+ 296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:14.603944 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:14.603949 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:14.603951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:14.604147 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:14.604421 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:14.604424 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:14.604426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:14.604509 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:14.604859 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:14.604862 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:14.604955 (+ 93us) client_negotiation.cc:769] Sending connection context
0926 16:42:14.605076 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 16:42:14.605212 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":24,"thread_start_us":83,"threads_started":1}
Sep 26 16:42:14 dist-test-slave-n1hw krb5kdc[23242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 16:42:14 dist-test-slave-n1hw krb5kdc[23242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:42:14.624624 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:14.616424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56602 (local address 127.21.12.126:33641)
0926 16:42:14.616552 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:14.616556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:14.616569 (+ 13us) server_negotiation.cc:407] Connection header received
0926 16:42:14.616608 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:14.616610 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:14.616648 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:14.616731 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:14.617395 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.618019 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.618845 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.619063 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.619143 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:14.621481 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:14.621484 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:14.621486 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:14.621509 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:14.623089 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:14.623457 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:14.623459 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:14.623461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:14.623534 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:14.623765 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:14.623768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:14.623769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:14.623929 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:14.624032 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:14.624300 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 16:42:14.624390 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 16:42:14.627421 23286 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56602:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:42:14.630581 23286 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:42:14.645026 23789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:14.639448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:40961 (local address 127.0.0.1:59650)
0926 16:42:14.639703 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:14.639718 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:14.639803 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:14.640808 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:14.640814 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:14.640835 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:14.640963 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:14.640970 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.642309 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.642316 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:14.643530 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:14.643539 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.644466 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.644473 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:14.644592 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:14.644637 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 16:42:14.644755 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 16:42:14.644876 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":78,"threads_started":1}
I20260926 16:42:14.645040 23791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:14.639903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59650 (local address 127.21.12.66:40961)
0926 16:42:14.640302 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:14.640315 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:14.640342 (+ 27us) server_negotiation.cc:407] Connection header received
0926 16:42:14.640415 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:14.640419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:14.640625 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:14.641133 (+ 508us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:14.641281 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.642201 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.643656 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.644350 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.644448 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:14.644595 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:14.644795 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 16:42:14.644882 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":98,"threads_started":1}
I20260926 16:42:14.645664 23787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:14.639019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.67:36983 (local address 127.0.0.1:42052)
0926 16:42:14.639532 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:14.639589 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:14.639733 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:14.640980 (+ 1247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:14.640984 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:14.641003 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:14.641100 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:14.641105 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.642616 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.642620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:14.643990 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:14.644001 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.645280 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.645286 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:14.645371 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:14.645388 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 16:42:14.645479 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 16:42:14.645544 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":80,"threads_started":1}
I20260926 16:42:14.646121 23788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:14.639167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42052 (local address 127.21.12.67:36983)
0926 16:42:14.640117 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:14.640130 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:14.640156 (+ 26us) server_negotiation.cc:407] Connection header received
0926 16:42:14.640278 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:14.640284 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:14.640517 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:14.640698 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:14.641233 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.642475 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.644163 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.644929 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.645025 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:14.645217 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:14.645510 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 16:42:14.645609 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":816,"thread_start_us":123,"threads_started":1}
I20260926 16:42:14.646227 23790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:14.639738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:39905 (local address 127.0.0.1:34630)
0926 16:42:14.640539 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:14.640552 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:14.640631 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:14.641410 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:14.641412 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:14.641428 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:14.641507 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:14.641512 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.643122 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.643126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:14.644418 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:14.644432 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.645832 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.645836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:14.645934 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:14.645958 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 16:42:14.646032 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 16:42:14.646099 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"thread_start_us":81,"threads_started":1}
I20260926 16:42:14.646636 23792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:14.640242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34630 (local address 127.21.12.65:39905)
0926 16:42:14.640860 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:14.640874 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:14.640895 (+ 21us) server_negotiation.cc:407] Connection header received
0926 16:42:14.640961 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:14.640966 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:14.641131 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:14.641278 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:14.641641 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.642904 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.644562 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.645456 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.645566 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:14.645744 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:14.646398 (+ 654us) server_negotiation.cc:299] Negotiation successful
0926 16:42:14.646493 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":66,"threads_started":1}
I20260926 16:42:14.648702 23704 tablet_service.cc:1467] Processing CreateTablet for tablet 1c230af54d2b409ab0895254b6432afd (DEFAULT_TABLE table=test-table [id=b50d85eec0aa44648361440c5a68c58c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:14.649063 23564 tablet_service.cc:1467] Processing CreateTablet for tablet 1c230af54d2b409ab0895254b6432afd (DEFAULT_TABLE table=test-table [id=b50d85eec0aa44648361440c5a68c58c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:14.649063 23424 tablet_service.cc:1467] Processing CreateTablet for tablet 1c230af54d2b409ab0895254b6432afd (DEFAULT_TABLE table=test-table [id=b50d85eec0aa44648361440c5a68c58c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:14.649765 23704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c230af54d2b409ab0895254b6432afd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:14.650151 23424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c230af54d2b409ab0895254b6432afd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:14.650151 23564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c230af54d2b409ab0895254b6432afd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:14.656363 23793 tablet_bootstrap.cc:492] T 1c230af54d2b409ab0895254b6432afd P ffedcedd84804834b6a74572c0b94cc8: Bootstrap starting.
I20260926 16:42:14.658192 23794 tablet_bootstrap.cc:492] T 1c230af54d2b409ab0895254b6432afd P 1a25832cf0b44dcbb487d80d238ed3f4: Bootstrap starting.
I20260926 16:42:14.659088 23793 tablet_bootstrap.cc:654] T 1c230af54d2b409ab0895254b6432afd P ffedcedd84804834b6a74572c0b94cc8: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:14.659243 23795 tablet_bootstrap.cc:492] T 1c230af54d2b409ab0895254b6432afd P 56c85d4ee801431f8c556a5dea08cf45: Bootstrap starting.
I20260926 16:42:14.660171 23793 log.cc:824] T 1c230af54d2b409ab0895254b6432afd P ffedcedd84804834b6a74572c0b94cc8: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:14.660497 23794 tablet_bootstrap.cc:654] T 1c230af54d2b409ab0895254b6432afd P 1a25832cf0b44dcbb487d80d238ed3f4: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:14.661331 23794 log.cc:824] T 1c230af54d2b409ab0895254b6432afd P 1a25832cf0b44dcbb487d80d238ed3f4: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:14.661419 23795 tablet_bootstrap.cc:654] T 1c230af54d2b409ab0895254b6432afd P 56c85d4ee801431f8c556a5dea08cf45: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:14.662282 23795 log.cc:824] T 1c230af54d2b409ab0895254b6432afd P 56c85d4ee801431f8c556a5dea08cf45: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:14.662616 23793 tablet_bootstrap.cc:492] T 1c230af54d2b409ab0895254b6432afd P ffedcedd84804834b6a74572c0b94cc8: No bootstrap required, opened a new log
I20260926 16:42:14.662832 23793 ts_tablet_manager.cc:1397] T 1c230af54d2b409ab0895254b6432afd P ffedcedd84804834b6a74572c0b94cc8: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 16:42:14.663128 23794 tablet_bootstrap.cc:492] T 1c230af54d2b409ab0895254b6432afd P 1a25832cf0b44dcbb487d80d238ed3f4: No bootstrap required, opened a new log
I20260926 16:42:14.663323 23794 ts_tablet_manager.cc:1397] T 1c230af54d2b409ab0895254b6432afd P 1a25832cf0b44dcbb487d80d238ed3f4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 16:42:14.664291 23795 tablet_bootstrap.cc:492] T 1c230af54d2b409ab0895254b6432afd P 56c85d4ee801431f8c556a5dea08cf45: No bootstrap required, opened a new log
I20260926 16:42:14.664474 23795 ts_tablet_manager.cc:1397] T 1c230af54d2b409ab0895254b6432afd P 56c85d4ee801431f8c556a5dea08cf45: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 16:42:14.666550 23794 raft_consensus.cc:348] T 1c230af54d2b409ab0895254b6432afd P 1a25832cf0b44dcbb487d80d238ed3f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } } peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } }
I20260926 16:42:14.666790 23794 raft_consensus.cc:374] T 1c230af54d2b409ab0895254b6432afd P 1a25832cf0b44dcbb487d80d238ed3f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:14.666842 23794 raft_consensus.cc:729] T 1c230af54d2b409ab0895254b6432afd P 1a25832cf0b44dcbb487d80d238ed3f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a25832cf0b44dcbb487d80d238ed3f4, State: Initialized, Role: FOLLOWER
I20260926 16:42:14.667338 23794 consensus_queue.cc:260] T 1c230af54d2b409ab0895254b6432afd P 1a25832cf0b44dcbb487d80d238ed3f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } } peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } }
I20260926 16:42:14.667322 23793 raft_consensus.cc:348] T 1c230af54d2b409ab0895254b6432afd P ffedcedd84804834b6a74572c0b94cc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } } peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } }
I20260926 16:42:14.667577 23793 raft_consensus.cc:374] T 1c230af54d2b409ab0895254b6432afd P ffedcedd84804834b6a74572c0b94cc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:14.667630 23793 raft_consensus.cc:729] T 1c230af54d2b409ab0895254b6432afd P ffedcedd84804834b6a74572c0b94cc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffedcedd84804834b6a74572c0b94cc8, State: Initialized, Role: FOLLOWER
I20260926 16:42:14.668057 23793 consensus_queue.cc:260] T 1c230af54d2b409ab0895254b6432afd P ffedcedd84804834b6a74572c0b94cc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } } peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } }
I20260926 16:42:14.668321 23794 ts_tablet_manager.cc:1428] T 1c230af54d2b409ab0895254b6432afd P 1a25832cf0b44dcbb487d80d238ed3f4: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 16:42:14.668689 23490 heartbeater.cc:503] Master 127.21.12.126:33641 was elected leader, sending a full tablet report...
I20260926 16:42:14.668710 23770 heartbeater.cc:503] Master 127.21.12.126:33641 was elected leader, sending a full tablet report...
I20260926 16:42:14.669092 23793 ts_tablet_manager.cc:1428] T 1c230af54d2b409ab0895254b6432afd P ffedcedd84804834b6a74572c0b94cc8: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 16:42:14.669646 23795 raft_consensus.cc:348] T 1c230af54d2b409ab0895254b6432afd P 56c85d4ee801431f8c556a5dea08cf45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } } peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } }
I20260926 16:42:14.669929 23795 raft_consensus.cc:374] T 1c230af54d2b409ab0895254b6432afd P 56c85d4ee801431f8c556a5dea08cf45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:14.669991 23795 raft_consensus.cc:729] T 1c230af54d2b409ab0895254b6432afd P 56c85d4ee801431f8c556a5dea08cf45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56c85d4ee801431f8c556a5dea08cf45, State: Initialized, Role: FOLLOWER
I20260926 16:42:14.670404 23795 consensus_queue.cc:260] T 1c230af54d2b409ab0895254b6432afd P 56c85d4ee801431f8c556a5dea08cf45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } } peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } }
I20260926 16:42:14.671115 23630 heartbeater.cc:503] Master 127.21.12.126:33641 was elected leader, sending a full tablet report...
I20260926 16:42:14.671492 23795 ts_tablet_manager.cc:1428] T 1c230af54d2b409ab0895254b6432afd P 56c85d4ee801431f8c556a5dea08cf45: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260926 16:42:14.805096 23801 raft_consensus.cc:482] T 1c230af54d2b409ab0895254b6432afd P 56c85d4ee801431f8c556a5dea08cf45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:42:14.805372 23801 raft_consensus.cc:504] T 1c230af54d2b409ab0895254b6432afd P 56c85d4ee801431f8c556a5dea08cf45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } } peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } }
I20260926 16:42:14.806596 23801 leader_election.cc:288] T 1c230af54d2b409ab0895254b6432afd P 56c85d4ee801431f8c556a5dea08cf45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffedcedd84804834b6a74572c0b94cc8 (127.21.12.67:36983), 1a25832cf0b44dcbb487d80d238ed3f4 (127.21.12.65:39905)
I20260926 16:42:14.810380 23788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:14.807006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:58561 (local address 127.21.12.67:36983)
0926 16:42:14.807131 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:14.807135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:14.807219 (+ 84us) server_negotiation.cc:407] Connection header received
0926 16:42:14.807413 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:14.807416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:14.807468 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:14.807542 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:14.807852 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.808347 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.809374 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.809833 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.809956 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:14.810007 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:14.810195 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 16:42:14.810259 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 16:42:14.810380 23523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:14.806933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.67:36983 (local address 127.21.12.66:58561)
0926 16:42:14.807078 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:14.807096 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:14.807291 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:14.807567 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:14.807570 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:14.807600 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:14.807725 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:14.807732 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.808455 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.808461 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:14.809254 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:14.809262 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.809945 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.809949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:14.810058 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:14.810087 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 16:42:14.810170 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 16:42:14.810252 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":3}
I20260926 16:42:14.810971 23724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c230af54d2b409ab0895254b6432afd" candidate_uuid: "56c85d4ee801431f8c556a5dea08cf45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffedcedd84804834b6a74572c0b94cc8" is_pre_election: true
I20260926 16:42:14.811194 23792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:14.807323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:46737 (local address 127.21.12.65:39905)
0926 16:42:14.807466 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:14.807471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:14.807745 (+ 274us) server_negotiation.cc:407] Connection header received
0926 16:42:14.807854 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:14.807857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:14.807906 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:14.807986 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:14.808587 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.809093 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.810155 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.810664 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.810786 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:14.810833 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:14.811020 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 16:42:14.811085 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 16:42:14.811208 23802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:14.807219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:39905 (local address 127.21.12.66:46737)
0926 16:42:14.807628 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:14.807642 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:14.807751 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:14.807995 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:14.807998 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:14.808013 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:14.808470 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:14.808476 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.809202 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.809207 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:14.810021 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:14.810030 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.810799 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.810803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:14.810903 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:14.810920 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 16:42:14.811006 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 16:42:14.811069 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":144,"threads_started":1}
I20260926 16:42:14.811282 23724 raft_consensus.cc:2454] T 1c230af54d2b409ab0895254b6432afd P ffedcedd84804834b6a74572c0b94cc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56c85d4ee801431f8c556a5dea08cf45 in term 0.
I20260926 16:42:14.811767 23516 leader_election.cc:302] T 1c230af54d2b409ab0895254b6432afd P 56c85d4ee801431f8c556a5dea08cf45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56c85d4ee801431f8c556a5dea08cf45, ffedcedd84804834b6a74572c0b94cc8; no voters:
I20260926 16:42:14.811810 23444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c230af54d2b409ab0895254b6432afd" candidate_uuid: "56c85d4ee801431f8c556a5dea08cf45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" is_pre_election: true
I20260926 16:42:14.812037 23801 raft_consensus.cc:2781] T 1c230af54d2b409ab0895254b6432afd P 56c85d4ee801431f8c556a5dea08cf45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:42:14.812062 23444 raft_consensus.cc:2454] T 1c230af54d2b409ab0895254b6432afd P 1a25832cf0b44dcbb487d80d238ed3f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56c85d4ee801431f8c556a5dea08cf45 in term 0.
I20260926 16:42:14.812168 23801 raft_consensus.cc:482] T 1c230af54d2b409ab0895254b6432afd P 56c85d4ee801431f8c556a5dea08cf45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:42:14.812227 23801 raft_consensus.cc:3037] T 1c230af54d2b409ab0895254b6432afd P 56c85d4ee801431f8c556a5dea08cf45 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:14.813262 23801 raft_consensus.cc:504] T 1c230af54d2b409ab0895254b6432afd P 56c85d4ee801431f8c556a5dea08cf45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } } peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } }
I20260926 16:42:14.813576 23801 leader_election.cc:288] T 1c230af54d2b409ab0895254b6432afd P 56c85d4ee801431f8c556a5dea08cf45 [CANDIDATE]: Term 1 election: Requested vote from peers ffedcedd84804834b6a74572c0b94cc8 (127.21.12.67:36983), 1a25832cf0b44dcbb487d80d238ed3f4 (127.21.12.65:39905)
I20260926 16:42:14.813908 23724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c230af54d2b409ab0895254b6432afd" candidate_uuid: "56c85d4ee801431f8c556a5dea08cf45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffedcedd84804834b6a74572c0b94cc8"
I20260926 16:42:14.814014 23444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c230af54d2b409ab0895254b6432afd" candidate_uuid: "56c85d4ee801431f8c556a5dea08cf45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a25832cf0b44dcbb487d80d238ed3f4"
I20260926 16:42:14.814038 23724 raft_consensus.cc:3037] T 1c230af54d2b409ab0895254b6432afd P ffedcedd84804834b6a74572c0b94cc8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:14.814128 23444 raft_consensus.cc:3037] T 1c230af54d2b409ab0895254b6432afd P 1a25832cf0b44dcbb487d80d238ed3f4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:14.815133 23724 raft_consensus.cc:2454] T 1c230af54d2b409ab0895254b6432afd P ffedcedd84804834b6a74572c0b94cc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56c85d4ee801431f8c556a5dea08cf45 in term 1.
I20260926 16:42:14.815261 23444 raft_consensus.cc:2454] T 1c230af54d2b409ab0895254b6432afd P 1a25832cf0b44dcbb487d80d238ed3f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56c85d4ee801431f8c556a5dea08cf45 in term 1.
I20260926 16:42:14.815491 23516 leader_election.cc:302] T 1c230af54d2b409ab0895254b6432afd P 56c85d4ee801431f8c556a5dea08cf45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56c85d4ee801431f8c556a5dea08cf45, ffedcedd84804834b6a74572c0b94cc8; no voters:
I20260926 16:42:14.815694 23801 raft_consensus.cc:2781] T 1c230af54d2b409ab0895254b6432afd P 56c85d4ee801431f8c556a5dea08cf45 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:14.815896 23801 raft_consensus.cc:686] T 1c230af54d2b409ab0895254b6432afd P 56c85d4ee801431f8c556a5dea08cf45 [term 1 LEADER]: Becoming Leader. State: Replica: 56c85d4ee801431f8c556a5dea08cf45, State: Running, Role: LEADER
I20260926 16:42:14.816239 23801 consensus_queue.cc:237] T 1c230af54d2b409ab0895254b6432afd P 56c85d4ee801431f8c556a5dea08cf45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } } peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } }
I20260926 16:42:14.819280 23286 catalog_manager.cc:5495] T 1c230af54d2b409ab0895254b6432afd P 56c85d4ee801431f8c556a5dea08cf45 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56c85d4ee801431f8c556a5dea08cf45 (127.21.12.66). New cstate: current_term: 1 leader_uuid: "56c85d4ee801431f8c556a5dea08cf45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } health_report { overall_health: HEALTHY } } }
W20260926 16:42:14.842105 23771 tablet.cc:2369] T 1c230af54d2b409ab0895254b6432afd P ffedcedd84804834b6a74572c0b94cc8: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 16:42:14 dist-test-slave-n1hw krb5kdc[23242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440933, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:42:14.868054 23810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:14.859634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:33641 (local address 127.0.0.1:56606)
0926 16:42:14.859976 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:14.859997 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:14.860138 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:14.860434 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:14.860436 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:14.860889 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:14.860989 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:14.860995 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.861742 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.861745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:14.862323 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:14.862331 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.862758 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.862761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:14.862839 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:14.863363 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:14.863375 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:14.864802 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:14.866444 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:14.866455 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:14.866461 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:14.866704 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:14.866972 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:14.866977 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:14.866979 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:14.867057 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:14.867463 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:14.867471 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:14.867575 (+ 104us) client_negotiation.cc:769] Sending connection context
0926 16:42:14.867695 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 16:42:14.867850 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":109,"threads_started":1}
I20260926 16:42:14.868053 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:14.859725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56606 (local address 127.21.12.126:33641)
0926 16:42:14.859941 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:14.859946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:14.860073 (+ 127us) server_negotiation.cc:407] Connection header received
0926 16:42:14.860266 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:14.860269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:14.860317 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:14.860418 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:14.861105 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.861625 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.862446 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:14.862655 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:14.862759 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:14.864928 (+ 2169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:14.864932 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:14.864935 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:14.864964 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:14.866335 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:14.866802 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:14.866807 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:14.866809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:14.866883 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:14.867144 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:14.867193 (+ 49us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:14.867194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:14.867348 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:14.867453 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:14.867738 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 16:42:14.867920 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":87,"server-negotiator.queue_time_us":104}
I20260926 16:42:14.871726 23286 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:56606:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 16:42:14.872453 23286 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260926 16:42:14.874354 23631 tablet.cc:2369] T 1c230af54d2b409ab0895254b6432afd P 56c85d4ee801431f8c556a5dea08cf45: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:42:14.877589 23704 tablet_service.cc:1467] Processing CreateTablet for tablet d24752bcc10e4f8e89024c80e9110fd8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4d8699afa5f3410780bc59c017a6d196]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 16:42:14.877414 23424 tablet_service.cc:1467] Processing CreateTablet for tablet d24752bcc10e4f8e89024c80e9110fd8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4d8699afa5f3410780bc59c017a6d196]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 16:42:14.878185 23424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d24752bcc10e4f8e89024c80e9110fd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:14.878185 23704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d24752bcc10e4f8e89024c80e9110fd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:14.878495 23564 tablet_service.cc:1467] Processing CreateTablet for tablet d24752bcc10e4f8e89024c80e9110fd8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4d8699afa5f3410780bc59c017a6d196]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 16:42:14.879102 23564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d24752bcc10e4f8e89024c80e9110fd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:14.881080 23793 tablet_bootstrap.cc:492] T d24752bcc10e4f8e89024c80e9110fd8 P ffedcedd84804834b6a74572c0b94cc8: Bootstrap starting.
I20260926 16:42:14.881080 23794 tablet_bootstrap.cc:492] T d24752bcc10e4f8e89024c80e9110fd8 P 1a25832cf0b44dcbb487d80d238ed3f4: Bootstrap starting.
I20260926 16:42:14.882095 23794 tablet_bootstrap.cc:654] T d24752bcc10e4f8e89024c80e9110fd8 P 1a25832cf0b44dcbb487d80d238ed3f4: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:14.882095 23793 tablet_bootstrap.cc:654] T d24752bcc10e4f8e89024c80e9110fd8 P ffedcedd84804834b6a74572c0b94cc8: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:14.882781 23795 tablet_bootstrap.cc:492] T d24752bcc10e4f8e89024c80e9110fd8 P 56c85d4ee801431f8c556a5dea08cf45: Bootstrap starting.
I20260926 16:42:14.883775 23795 tablet_bootstrap.cc:654] T d24752bcc10e4f8e89024c80e9110fd8 P 56c85d4ee801431f8c556a5dea08cf45: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:14.883975 23793 tablet_bootstrap.cc:492] T d24752bcc10e4f8e89024c80e9110fd8 P ffedcedd84804834b6a74572c0b94cc8: No bootstrap required, opened a new log
I20260926 16:42:14.884045 23793 ts_tablet_manager.cc:1397] T d24752bcc10e4f8e89024c80e9110fd8 P ffedcedd84804834b6a74572c0b94cc8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 16:42:14.884125 23794 tablet_bootstrap.cc:492] T d24752bcc10e4f8e89024c80e9110fd8 P 1a25832cf0b44dcbb487d80d238ed3f4: No bootstrap required, opened a new log
I20260926 16:42:14.884193 23794 ts_tablet_manager.cc:1397] T d24752bcc10e4f8e89024c80e9110fd8 P 1a25832cf0b44dcbb487d80d238ed3f4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 16:42:14.884492 23793 raft_consensus.cc:348] T d24752bcc10e4f8e89024c80e9110fd8 P ffedcedd84804834b6a74572c0b94cc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } }
I20260926 16:42:14.884579 23793 raft_consensus.cc:374] T d24752bcc10e4f8e89024c80e9110fd8 P ffedcedd84804834b6a74572c0b94cc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:14.884599 23793 raft_consensus.cc:729] T d24752bcc10e4f8e89024c80e9110fd8 P ffedcedd84804834b6a74572c0b94cc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffedcedd84804834b6a74572c0b94cc8, State: Initialized, Role: FOLLOWER
I20260926 16:42:14.884589 23794 raft_consensus.cc:348] T d24752bcc10e4f8e89024c80e9110fd8 P 1a25832cf0b44dcbb487d80d238ed3f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } }
I20260926 16:42:14.884686 23794 raft_consensus.cc:374] T d24752bcc10e4f8e89024c80e9110fd8 P 1a25832cf0b44dcbb487d80d238ed3f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:14.884716 23794 raft_consensus.cc:729] T d24752bcc10e4f8e89024c80e9110fd8 P 1a25832cf0b44dcbb487d80d238ed3f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a25832cf0b44dcbb487d80d238ed3f4, State: Initialized, Role: FOLLOWER
I20260926 16:42:14.884745 23793 consensus_queue.cc:260] T d24752bcc10e4f8e89024c80e9110fd8 P ffedcedd84804834b6a74572c0b94cc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } }
I20260926 16:42:14.884827 23794 consensus_queue.cc:260] T d24752bcc10e4f8e89024c80e9110fd8 P 1a25832cf0b44dcbb487d80d238ed3f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } }
I20260926 16:42:14.885051 23794 ts_tablet_manager.cc:1428] T d24752bcc10e4f8e89024c80e9110fd8 P 1a25832cf0b44dcbb487d80d238ed3f4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 16:42:14.885072 23793 ts_tablet_manager.cc:1428] T d24752bcc10e4f8e89024c80e9110fd8 P ffedcedd84804834b6a74572c0b94cc8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 16:42:14.885022 23800 tablet_replica.cc:439] T d24752bcc10e4f8e89024c80e9110fd8 P ffedcedd84804834b6a74572c0b94cc8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } } }
I20260926 16:42:14.885152 23800 tablet_replica.cc:442] T d24752bcc10e4f8e89024c80e9110fd8 P ffedcedd84804834b6a74572c0b94cc8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:42:14.885484 23799 tablet_replica.cc:439] T d24752bcc10e4f8e89024c80e9110fd8 P 1a25832cf0b44dcbb487d80d238ed3f4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } } }
I20260926 16:42:14.885618 23799 tablet_replica.cc:442] T d24752bcc10e4f8e89024c80e9110fd8 P 1a25832cf0b44dcbb487d80d238ed3f4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:42:14.887348 23795 tablet_bootstrap.cc:492] T d24752bcc10e4f8e89024c80e9110fd8 P 56c85d4ee801431f8c556a5dea08cf45: No bootstrap required, opened a new log
I20260926 16:42:14.887459 23795 ts_tablet_manager.cc:1397] T d24752bcc10e4f8e89024c80e9110fd8 P 56c85d4ee801431f8c556a5dea08cf45: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260926 16:42:14.887868 23795 raft_consensus.cc:348] T d24752bcc10e4f8e89024c80e9110fd8 P 56c85d4ee801431f8c556a5dea08cf45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } }
I20260926 16:42:14.887969 23795 raft_consensus.cc:374] T d24752bcc10e4f8e89024c80e9110fd8 P 56c85d4ee801431f8c556a5dea08cf45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:14.888000 23795 raft_consensus.cc:729] T d24752bcc10e4f8e89024c80e9110fd8 P 56c85d4ee801431f8c556a5dea08cf45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56c85d4ee801431f8c556a5dea08cf45, State: Initialized, Role: FOLLOWER
I20260926 16:42:14.888114 23795 consensus_queue.cc:260] T d24752bcc10e4f8e89024c80e9110fd8 P 56c85d4ee801431f8c556a5dea08cf45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } }
I20260926 16:42:14.888372 23795 ts_tablet_manager.cc:1428] T d24752bcc10e4f8e89024c80e9110fd8 P 56c85d4ee801431f8c556a5dea08cf45: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 16:42:14.888322 23801 tablet_replica.cc:439] T d24752bcc10e4f8e89024c80e9110fd8 P 56c85d4ee801431f8c556a5dea08cf45: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } } }
I20260926 16:42:14.888435 23801 tablet_replica.cc:442] T d24752bcc10e4f8e89024c80e9110fd8 P 56c85d4ee801431f8c556a5dea08cf45: This TxnStatusTablet replica's current role is: FOLLOWER
W20260926 16:42:14.907122 23491 tablet.cc:2369] T 1c230af54d2b409ab0895254b6432afd P 1a25832cf0b44dcbb487d80d238ed3f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:42:15.171746 23800 raft_consensus.cc:482] T d24752bcc10e4f8e89024c80e9110fd8 P ffedcedd84804834b6a74572c0b94cc8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:42:15.171896 23800 raft_consensus.cc:504] T d24752bcc10e4f8e89024c80e9110fd8 P ffedcedd84804834b6a74572c0b94cc8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } }
I20260926 16:42:15.172919 23800 leader_election.cc:288] T d24752bcc10e4f8e89024c80e9110fd8 P ffedcedd84804834b6a74572c0b94cc8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a25832cf0b44dcbb487d80d238ed3f4 (127.21.12.65:39905), 56c85d4ee801431f8c556a5dea08cf45 (127.21.12.66:40961)
I20260926 16:42:15.176765 23792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:15.173365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:51345 (local address 127.21.12.65:39905)
0926 16:42:15.173492 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:15.173497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:15.173674 (+ 177us) server_negotiation.cc:407] Connection header received
0926 16:42:15.173855 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:15.173858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:15.173904 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:15.173986 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:15.174251 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:15.174746 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:15.175826 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:15.176217 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:15.176341 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:15.176401 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:15.176575 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 16:42:15.176644 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 16:42:15.176942 23815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:15.173382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:55711 (local address 127.21.12.66:40961)
0926 16:42:15.173630 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:15.173646 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:15.173927 (+ 281us) server_negotiation.cc:407] Connection header received
0926 16:42:15.174086 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:15.174089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:15.174130 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:15.174227 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:15.174478 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:15.174988 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:15.175960 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:15.176385 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:15.176509 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:15.176560 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:15.176738 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 16:42:15.176809 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":51,"threads_started":1}
I20260926 16:42:15.176785 23814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:15.173280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:39905 (local address 127.21.12.67:51345)
0926 16:42:15.173576 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:15.173588 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:15.173725 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:15.174000 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:15.174004 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:15.174021 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:15.174146 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:15.174151 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:15.174878 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:15.174881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:15.175702 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:15.175711 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:15.176336 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:15.176340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:15.176438 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:15.176461 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 16:42:15.176561 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 16:42:15.176639 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":80,"threads_started":1}
I20260926 16:42:15.176949 23816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:15.173280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:40961 (local address 127.21.12.67:55711)
0926 16:42:15.173834 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:15.173853 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:15.173969 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:15.174241 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:15.174244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:15.174259 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:15.174362 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:15.174368 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:15.175099 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:15.175105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:15.175838 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:15.175846 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:15.176509 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:15.176513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:15.176605 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:15.176626 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 16:42:15.176728 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 16:42:15.176796 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":234,"thread_start_us":46,"threads_started":1}
I20260926 16:42:15.177434 23444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24752bcc10e4f8e89024c80e9110fd8" candidate_uuid: "ffedcedd84804834b6a74572c0b94cc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" is_pre_election: true
I20260926 16:42:15.177623 23444 raft_consensus.cc:2454] T d24752bcc10e4f8e89024c80e9110fd8 P 1a25832cf0b44dcbb487d80d238ed3f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffedcedd84804834b6a74572c0b94cc8 in term 0.
I20260926 16:42:15.177567 23584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24752bcc10e4f8e89024c80e9110fd8" candidate_uuid: "ffedcedd84804834b6a74572c0b94cc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56c85d4ee801431f8c556a5dea08cf45" is_pre_election: true
I20260926 16:42:15.177831 23584 raft_consensus.cc:2454] T d24752bcc10e4f8e89024c80e9110fd8 P 56c85d4ee801431f8c556a5dea08cf45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffedcedd84804834b6a74572c0b94cc8 in term 0.
I20260926 16:42:15.177974 23656 leader_election.cc:302] T d24752bcc10e4f8e89024c80e9110fd8 P ffedcedd84804834b6a74572c0b94cc8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a25832cf0b44dcbb487d80d238ed3f4, ffedcedd84804834b6a74572c0b94cc8; no voters:
I20260926 16:42:15.178249 23800 raft_consensus.cc:2781] T d24752bcc10e4f8e89024c80e9110fd8 P ffedcedd84804834b6a74572c0b94cc8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:42:15.178325 23800 raft_consensus.cc:482] T d24752bcc10e4f8e89024c80e9110fd8 P ffedcedd84804834b6a74572c0b94cc8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:42:15.178383 23800 raft_consensus.cc:3037] T d24752bcc10e4f8e89024c80e9110fd8 P ffedcedd84804834b6a74572c0b94cc8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:15.179353 23800 raft_consensus.cc:504] T d24752bcc10e4f8e89024c80e9110fd8 P ffedcedd84804834b6a74572c0b94cc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } }
I20260926 16:42:15.179680 23800 leader_election.cc:288] T d24752bcc10e4f8e89024c80e9110fd8 P ffedcedd84804834b6a74572c0b94cc8 [CANDIDATE]: Term 1 election: Requested vote from peers 1a25832cf0b44dcbb487d80d238ed3f4 (127.21.12.65:39905), 56c85d4ee801431f8c556a5dea08cf45 (127.21.12.66:40961)
I20260926 16:42:15.180006 23444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24752bcc10e4f8e89024c80e9110fd8" candidate_uuid: "ffedcedd84804834b6a74572c0b94cc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a25832cf0b44dcbb487d80d238ed3f4"
I20260926 16:42:15.180013 23584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24752bcc10e4f8e89024c80e9110fd8" candidate_uuid: "ffedcedd84804834b6a74572c0b94cc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56c85d4ee801431f8c556a5dea08cf45"
I20260926 16:42:15.180136 23584 raft_consensus.cc:3037] T d24752bcc10e4f8e89024c80e9110fd8 P 56c85d4ee801431f8c556a5dea08cf45 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:15.180137 23444 raft_consensus.cc:3037] T d24752bcc10e4f8e89024c80e9110fd8 P 1a25832cf0b44dcbb487d80d238ed3f4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:15.180971 23444 raft_consensus.cc:2454] T d24752bcc10e4f8e89024c80e9110fd8 P 1a25832cf0b44dcbb487d80d238ed3f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffedcedd84804834b6a74572c0b94cc8 in term 1.
I20260926 16:42:15.181031 23584 raft_consensus.cc:2454] T d24752bcc10e4f8e89024c80e9110fd8 P 56c85d4ee801431f8c556a5dea08cf45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffedcedd84804834b6a74572c0b94cc8 in term 1.
I20260926 16:42:15.181299 23657 leader_election.cc:302] T d24752bcc10e4f8e89024c80e9110fd8 P ffedcedd84804834b6a74572c0b94cc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1a25832cf0b44dcbb487d80d238ed3f4, 56c85d4ee801431f8c556a5dea08cf45, ffedcedd84804834b6a74572c0b94cc8; no voters:
I20260926 16:42:15.181501 23800 raft_consensus.cc:2781] T d24752bcc10e4f8e89024c80e9110fd8 P ffedcedd84804834b6a74572c0b94cc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:15.181708 23800 raft_consensus.cc:686] T d24752bcc10e4f8e89024c80e9110fd8 P ffedcedd84804834b6a74572c0b94cc8 [term 1 LEADER]: Becoming Leader. State: Replica: ffedcedd84804834b6a74572c0b94cc8, State: Running, Role: LEADER
I20260926 16:42:15.182042 23800 consensus_queue.cc:237] T d24752bcc10e4f8e89024c80e9110fd8 P ffedcedd84804834b6a74572c0b94cc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } }
I20260926 16:42:15.184039 23817 tablet_replica.cc:439] T d24752bcc10e4f8e89024c80e9110fd8 P ffedcedd84804834b6a74572c0b94cc8: TxnStatusTablet state changed. Reason: New leader ffedcedd84804834b6a74572c0b94cc8. Latest consensus state: current_term: 1 leader_uuid: "ffedcedd84804834b6a74572c0b94cc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } } }
I20260926 16:42:15.184144 23817 tablet_replica.cc:442] T d24752bcc10e4f8e89024c80e9110fd8 P ffedcedd84804834b6a74572c0b94cc8: This TxnStatusTablet replica's current role is: LEADER
I20260926 16:42:15.184401 23819 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 16:42:15.184597 23819 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 16:42:15.184988 23286 catalog_manager.cc:5495] T d24752bcc10e4f8e89024c80e9110fd8 P ffedcedd84804834b6a74572c0b94cc8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ffedcedd84804834b6a74572c0b94cc8 (127.21.12.67). New cstate: current_term: 1 leader_uuid: "ffedcedd84804834b6a74572c0b94cc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } health_report { overall_health: HEALTHY } } }
I20260926 16:42:15.250629 23788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:15.246658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42064 (local address 127.21.12.67:36983)
0926 16:42:15.246812 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:15.246818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:15.246834 (+ 16us) server_negotiation.cc:407] Connection header received
0926 16:42:15.246917 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:15.246920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:15.246982 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:15.247087 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:42:15.247342 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:15.248122 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:15.248969 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:15.249321 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:15.249424 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:15.249691 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:42:15.250248 (+ 557us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:42:15.250365 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:15.250436 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 16:42:15.250495 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 16:42:15.250645 23810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:15.246144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.67:36983 (local address 127.0.0.1:42064)
0926 16:42:15.246357 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:15.246370 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:15.246465 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:15.247125 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:15.247128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:15.247135 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 16:42:15.247227 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:15.247231 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:15.248239 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:15.248242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:15.248851 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:15.248859 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:15.249430 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:15.249432 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:15.249499 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:15.249551 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 16:42:15.250350 (+ 799us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 16:42:15.250364 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 16:42:15.250423 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 16:42:15.250489 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 16:42:15.255810 23444 raft_consensus.cc:1261] T d24752bcc10e4f8e89024c80e9110fd8 P 1a25832cf0b44dcbb487d80d238ed3f4 [term 1 FOLLOWER]: Refusing update from remote peer ffedcedd84804834b6a74572c0b94cc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:42:15.255821 23584 raft_consensus.cc:1261] T d24752bcc10e4f8e89024c80e9110fd8 P 56c85d4ee801431f8c556a5dea08cf45 [term 1 FOLLOWER]: Refusing update from remote peer ffedcedd84804834b6a74572c0b94cc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:42:15.256460 23817 consensus_queue.cc:1035] T d24752bcc10e4f8e89024c80e9110fd8 P ffedcedd84804834b6a74572c0b94cc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:42:15.256759 23800 consensus_queue.cc:1035] T d24752bcc10e4f8e89024c80e9110fd8 P ffedcedd84804834b6a74572c0b94cc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:42:15.260003 23801 tablet_replica.cc:439] T d24752bcc10e4f8e89024c80e9110fd8 P 56c85d4ee801431f8c556a5dea08cf45: TxnStatusTablet state changed. Reason: New leader ffedcedd84804834b6a74572c0b94cc8. Latest consensus state: current_term: 1 leader_uuid: "ffedcedd84804834b6a74572c0b94cc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } } }
I20260926 16:42:15.260128 23801 tablet_replica.cc:442] T d24752bcc10e4f8e89024c80e9110fd8 P 56c85d4ee801431f8c556a5dea08cf45: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:42:15.260697 23799 tablet_replica.cc:439] T d24752bcc10e4f8e89024c80e9110fd8 P 1a25832cf0b44dcbb487d80d238ed3f4: TxnStatusTablet state changed. Reason: New leader ffedcedd84804834b6a74572c0b94cc8. Latest consensus state: current_term: 1 leader_uuid: "ffedcedd84804834b6a74572c0b94cc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } } }
I20260926 16:42:15.260805 23799 tablet_replica.cc:442] T d24752bcc10e4f8e89024c80e9110fd8 P 1a25832cf0b44dcbb487d80d238ed3f4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:42:15.262730 23800 tablet_replica.cc:439] T d24752bcc10e4f8e89024c80e9110fd8 P ffedcedd84804834b6a74572c0b94cc8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ffedcedd84804834b6a74572c0b94cc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } } }
I20260926 16:42:15.262833 23800 tablet_replica.cc:442] T d24752bcc10e4f8e89024c80e9110fd8 P ffedcedd84804834b6a74572c0b94cc8: This TxnStatusTablet replica's current role is: LEADER
I20260926 16:42:15.264077 23822 mvcc.cc:205] Tried to move back new op lower bound from 7333646070801088512 to 7333646070507819008. Current Snapshot: MvccSnapshot[applied={T|T < 7333646070801088512}]
I20260926 16:42:15.264460 23801 tablet_replica.cc:439] T d24752bcc10e4f8e89024c80e9110fd8 P 56c85d4ee801431f8c556a5dea08cf45: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ffedcedd84804834b6a74572c0b94cc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } } }
I20260926 16:42:15.264568 23801 tablet_replica.cc:442] T d24752bcc10e4f8e89024c80e9110fd8 P 56c85d4ee801431f8c556a5dea08cf45: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:42:15.264582 23824 mvcc.cc:205] Tried to move back new op lower bound from 7333646070801088512 to 7333646070507819008. Current Snapshot: MvccSnapshot[applied={T|T < 7333646070801088512}]
I20260926 16:42:15.265223 23817 tablet_replica.cc:439] T d24752bcc10e4f8e89024c80e9110fd8 P ffedcedd84804834b6a74572c0b94cc8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ffedcedd84804834b6a74572c0b94cc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } } }
I20260926 16:42:15.265324 23817 tablet_replica.cc:442] T d24752bcc10e4f8e89024c80e9110fd8 P ffedcedd84804834b6a74572c0b94cc8: This TxnStatusTablet replica's current role is: LEADER
I20260926 16:42:15.266396 23799 tablet_replica.cc:439] T d24752bcc10e4f8e89024c80e9110fd8 P 1a25832cf0b44dcbb487d80d238ed3f4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ffedcedd84804834b6a74572c0b94cc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } } }
I20260926 16:42:15.266505 23799 tablet_replica.cc:442] T d24752bcc10e4f8e89024c80e9110fd8 P 1a25832cf0b44dcbb487d80d238ed3f4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:42:15.266896 23823 mvcc.cc:205] Tried to move back new op lower bound from 7333646070801088512 to 7333646070507819008. Current Snapshot: MvccSnapshot[applied={T|T < 7333646070801088512}]
I20260926 16:42:15.277415 23815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:15.271039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59666 (local address 127.21.12.66:40961)
0926 16:42:15.271183 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:15.271188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:15.274082 (+ 2894us) server_negotiation.cc:407] Connection header received
0926 16:42:15.274197 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:15.274200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:15.274260 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:15.274365 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:42:15.274613 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:15.275122 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:15.275994 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:15.276235 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:15.276346 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:15.276628 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:42:15.277032 (+ 404us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:42:15.277177 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:15.277239 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 16:42:15.277290 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 16:42:15.288141 23788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:15.284396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:40937 (local address 127.21.12.67:36983)
0926 16:42:15.284509 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:15.284515 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:15.284765 (+ 250us) server_negotiation.cc:407] Connection header received
0926 16:42:15.284885 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:15.284888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:15.284930 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:15.284992 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:42:15.285251 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:15.285777 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:15.287050 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:15.287276 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:15.287355 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:15.287719 (+ 364us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:42:15.287783 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:42:15.287839 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:15.287965 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 16:42:15.288008 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 16:42:15.288187 23834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:15.284246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.67:36983 (local address 127.21.12.66:40937)
0926 16:42:15.284667 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:15.284681 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:15.284787 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:15.285025 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:15.285028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:15.285036 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 16:42:15.285151 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:15.285156 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:15.285887 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:15.285891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:15.286908 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:15.286920 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:15.287405 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:15.287409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:15.287509 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:15.287558 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 16:42:15.287869 (+ 311us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 16:42:15.287875 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 16:42:15.287942 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 16:42:15.288020 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":125,"threads_started":1}
I20260926 16:42:15.294795 23444 raft_consensus.cc:1261] T 1c230af54d2b409ab0895254b6432afd P 1a25832cf0b44dcbb487d80d238ed3f4 [term 1 FOLLOWER]: Refusing update from remote peer 56c85d4ee801431f8c556a5dea08cf45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:42:15.295049 23724 raft_consensus.cc:1261] T 1c230af54d2b409ab0895254b6432afd P ffedcedd84804834b6a74572c0b94cc8 [term 1 FOLLOWER]: Refusing update from remote peer 56c85d4ee801431f8c556a5dea08cf45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:42:15.295233 23801 consensus_queue.cc:1035] T 1c230af54d2b409ab0895254b6432afd P 56c85d4ee801431f8c556a5dea08cf45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:42:15.295554 23803 consensus_queue.cc:1035] T 1c230af54d2b409ab0895254b6432afd P 56c85d4ee801431f8c556a5dea08cf45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:42:15.318521 23815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:15.314494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:53963 (local address 127.21.12.66:40961)
0926 16:42:15.314628 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:15.314633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:15.314886 (+ 253us) server_negotiation.cc:407] Connection header received
0926 16:42:15.314999 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:15.315001 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:15.315107 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:15.315183 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:42:15.315470 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:15.315972 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:15.317328 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:15.317624 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:15.317731 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:15.318110 (+ 379us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:42:15.318184 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:42:15.318254 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:15.318377 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 16:42:15.318419 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 16:42:15.318624 23840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:15.314423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:40961 (local address 127.21.12.67:53963)
0926 16:42:15.314778 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:15.314794 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:15.314906 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:15.315200 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:15.315204 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:15.315213 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 16:42:15.315353 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:15.315361 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:15.316095 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:15.316099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:15.317167 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:15.317178 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:15.317807 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:15.317813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:15.317908 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:15.317959 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 16:42:15.318284 (+ 325us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 16:42:15.318291 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 16:42:15.318371 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 16:42:15.318453 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":114,"threads_started":1}
I20260926 16:42:15.347218 23287 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:42:15.347419 23287 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:42:15.350118 23287 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 549ff0b7deb048668009a29152477376: Sending DeleteTablet for 3 replicas of tablet 1c230af54d2b409ab0895254b6432afd
I20260926 16:42:15.350924 23564 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c230af54d2b409ab0895254b6432afd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:15 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:59650
I20260926 16:42:15.350960 23424 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c230af54d2b409ab0895254b6432afd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:15 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:34630
I20260926 16:42:15.351401 23844 tablet_replica.cc:330] T 1c230af54d2b409ab0895254b6432afd P 1a25832cf0b44dcbb487d80d238ed3f4: stopping tablet replica
I20260926 16:42:15.351649 23704 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c230af54d2b409ab0895254b6432afd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:15 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:42052
I20260926 16:42:15.351780 23844 raft_consensus.cc:2229] T 1c230af54d2b409ab0895254b6432afd P 1a25832cf0b44dcbb487d80d238ed3f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:42:15.352111 23844 raft_consensus.cc:2258] T 1c230af54d2b409ab0895254b6432afd P 1a25832cf0b44dcbb487d80d238ed3f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:42:15.352124 23845 tablet_replica.cc:330] T 1c230af54d2b409ab0895254b6432afd P ffedcedd84804834b6a74572c0b94cc8: stopping tablet replica
I20260926 16:42:15.352092 23287 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56602:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 16:42:15.352367 23845 raft_consensus.cc:2229] T 1c230af54d2b409ab0895254b6432afd P ffedcedd84804834b6a74572c0b94cc8 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260926 16:42:15.352501 23287 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:42:15.352527 23845 raft_consensus.cc:2258] T 1c230af54d2b409ab0895254b6432afd P ffedcedd84804834b6a74572c0b94cc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:42:15.353833 23845 ts_tablet_manager.cc:1905] T 1c230af54d2b409ab0895254b6432afd P ffedcedd84804834b6a74572c0b94cc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:42:15.353986 23844 ts_tablet_manager.cc:1905] T 1c230af54d2b409ab0895254b6432afd P 1a25832cf0b44dcbb487d80d238ed3f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:42:15.356652 23845 ts_tablet_manager.cc:1918] T 1c230af54d2b409ab0895254b6432afd P ffedcedd84804834b6a74572c0b94cc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:42:15.356765 23845 log.cc:1196] T 1c230af54d2b409ab0895254b6432afd P ffedcedd84804834b6a74572c0b94cc8: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-2/wal/wals/1c230af54d2b409ab0895254b6432afd
I20260926 16:42:15.357085 23845 ts_tablet_manager.cc:1939] T 1c230af54d2b409ab0895254b6432afd P ffedcedd84804834b6a74572c0b94cc8: Deleting consensus metadata
I20260926 16:42:15.358312 23424 tablet_service.cc:1467] Processing CreateTablet for tablet 2ff1f1f69d3f40858cb1ff05722da6f2 (DEFAULT_TABLE table=test-table [id=283e783aac8c4aa9b029e286430bfb3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:15.358399 23704 tablet_service.cc:1467] Processing CreateTablet for tablet 2ff1f1f69d3f40858cb1ff05722da6f2 (DEFAULT_TABLE table=test-table [id=283e783aac8c4aa9b029e286430bfb3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:15.358595 23424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ff1f1f69d3f40858cb1ff05722da6f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:15.358608 23704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ff1f1f69d3f40858cb1ff05722da6f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:15.358728 23843 tablet_replica.cc:330] T 1c230af54d2b409ab0895254b6432afd P 56c85d4ee801431f8c556a5dea08cf45: stopping tablet replica
I20260926 16:42:15.359017 23843 raft_consensus.cc:2229] T 1c230af54d2b409ab0895254b6432afd P 56c85d4ee801431f8c556a5dea08cf45 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:42:15.359176 23844 ts_tablet_manager.cc:1918] T 1c230af54d2b409ab0895254b6432afd P 1a25832cf0b44dcbb487d80d238ed3f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:42:15.359301 23844 log.cc:1196] T 1c230af54d2b409ab0895254b6432afd P 1a25832cf0b44dcbb487d80d238ed3f4: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-0/wal/wals/1c230af54d2b409ab0895254b6432afd
I20260926 16:42:15.359642 23844 ts_tablet_manager.cc:1939] T 1c230af54d2b409ab0895254b6432afd P 1a25832cf0b44dcbb487d80d238ed3f4: Deleting consensus metadata
I20260926 16:42:15.360015 23843 raft_consensus.cc:2258] T 1c230af54d2b409ab0895254b6432afd P 56c85d4ee801431f8c556a5dea08cf45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:42:15.360715 23794 tablet_bootstrap.cc:492] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 1a25832cf0b44dcbb487d80d238ed3f4: Bootstrap starting.
I20260926 16:42:15.360728 23793 tablet_bootstrap.cc:492] T 2ff1f1f69d3f40858cb1ff05722da6f2 P ffedcedd84804834b6a74572c0b94cc8: Bootstrap starting.
I20260926 16:42:15.361542 23793 tablet_bootstrap.cc:654] T 2ff1f1f69d3f40858cb1ff05722da6f2 P ffedcedd84804834b6a74572c0b94cc8: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:15.361562 23794 tablet_bootstrap.cc:654] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 1a25832cf0b44dcbb487d80d238ed3f4: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:15.362289 23564 tablet_service.cc:1467] Processing CreateTablet for tablet 2ff1f1f69d3f40858cb1ff05722da6f2 (DEFAULT_TABLE table=test-table [id=283e783aac8c4aa9b029e286430bfb3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:15.362448 23843 ts_tablet_manager.cc:1905] T 1c230af54d2b409ab0895254b6432afd P 56c85d4ee801431f8c556a5dea08cf45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:42:15.362437 23272 catalog_manager.cc:4841] TS ffedcedd84804834b6a74572c0b94cc8 (127.21.12.67:36983): tablet 1c230af54d2b409ab0895254b6432afd (table test-table [id=b50d85eec0aa44648361440c5a68c58c]) successfully deleted
I20260926 16:42:15.362569 23564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ff1f1f69d3f40858cb1ff05722da6f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:15.362721 23272 catalog_manager.cc:4841] TS 1a25832cf0b44dcbb487d80d238ed3f4 (127.21.12.65:39905): tablet 1c230af54d2b409ab0895254b6432afd (table test-table [id=b50d85eec0aa44648361440c5a68c58c]) successfully deleted
I20260926 16:42:15.363402 23794 tablet_bootstrap.cc:492] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 1a25832cf0b44dcbb487d80d238ed3f4: No bootstrap required, opened a new log
I20260926 16:42:15.363487 23794 ts_tablet_manager.cc:1397] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 1a25832cf0b44dcbb487d80d238ed3f4: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260926 16:42:15.363919 23794 raft_consensus.cc:348] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 1a25832cf0b44dcbb487d80d238ed3f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } }
I20260926 16:42:15.364050 23794 raft_consensus.cc:374] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 1a25832cf0b44dcbb487d80d238ed3f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:15.364091 23794 raft_consensus.cc:729] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 1a25832cf0b44dcbb487d80d238ed3f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a25832cf0b44dcbb487d80d238ed3f4, State: Initialized, Role: FOLLOWER
I20260926 16:42:15.364204 23794 consensus_queue.cc:260] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 1a25832cf0b44dcbb487d80d238ed3f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } }
I20260926 16:42:15.364468 23794 ts_tablet_manager.cc:1428] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 1a25832cf0b44dcbb487d80d238ed3f4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 16:42:15.365319 23795 tablet_bootstrap.cc:492] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 56c85d4ee801431f8c556a5dea08cf45: Bootstrap starting.
I20260926 16:42:15.365633 23843 ts_tablet_manager.cc:1918] T 1c230af54d2b409ab0895254b6432afd P 56c85d4ee801431f8c556a5dea08cf45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:42:15.365797 23843 log.cc:1196] T 1c230af54d2b409ab0895254b6432afd P 56c85d4ee801431f8c556a5dea08cf45: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-1/wal/wals/1c230af54d2b409ab0895254b6432afd
I20260926 16:42:15.366144 23843 ts_tablet_manager.cc:1939] T 1c230af54d2b409ab0895254b6432afd P 56c85d4ee801431f8c556a5dea08cf45: Deleting consensus metadata
I20260926 16:42:15.366242 23795 tablet_bootstrap.cc:654] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 56c85d4ee801431f8c556a5dea08cf45: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:15.367225 23273 catalog_manager.cc:4841] TS 56c85d4ee801431f8c556a5dea08cf45 (127.21.12.66:40961): tablet 1c230af54d2b409ab0895254b6432afd (table test-table [id=b50d85eec0aa44648361440c5a68c58c]) successfully deleted
I20260926 16:42:15.367822 23793 tablet_bootstrap.cc:492] T 2ff1f1f69d3f40858cb1ff05722da6f2 P ffedcedd84804834b6a74572c0b94cc8: No bootstrap required, opened a new log
I20260926 16:42:15.367913 23793 ts_tablet_manager.cc:1397] T 2ff1f1f69d3f40858cb1ff05722da6f2 P ffedcedd84804834b6a74572c0b94cc8: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260926 16:42:15.368103 23795 tablet_bootstrap.cc:492] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 56c85d4ee801431f8c556a5dea08cf45: No bootstrap required, opened a new log
I20260926 16:42:15.368184 23795 ts_tablet_manager.cc:1397] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 56c85d4ee801431f8c556a5dea08cf45: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 16:42:15.368350 23793 raft_consensus.cc:348] T 2ff1f1f69d3f40858cb1ff05722da6f2 P ffedcedd84804834b6a74572c0b94cc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } }
I20260926 16:42:15.368455 23793 raft_consensus.cc:374] T 2ff1f1f69d3f40858cb1ff05722da6f2 P ffedcedd84804834b6a74572c0b94cc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:15.368489 23793 raft_consensus.cc:729] T 2ff1f1f69d3f40858cb1ff05722da6f2 P ffedcedd84804834b6a74572c0b94cc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffedcedd84804834b6a74572c0b94cc8, State: Initialized, Role: FOLLOWER
I20260926 16:42:15.368598 23793 consensus_queue.cc:260] T 2ff1f1f69d3f40858cb1ff05722da6f2 P ffedcedd84804834b6a74572c0b94cc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } }
I20260926 16:42:15.368633 23795 raft_consensus.cc:348] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 56c85d4ee801431f8c556a5dea08cf45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } }
I20260926 16:42:15.368728 23795 raft_consensus.cc:374] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 56c85d4ee801431f8c556a5dea08cf45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:15.368770 23795 raft_consensus.cc:729] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 56c85d4ee801431f8c556a5dea08cf45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56c85d4ee801431f8c556a5dea08cf45, State: Initialized, Role: FOLLOWER
I20260926 16:42:15.368902 23793 ts_tablet_manager.cc:1428] T 2ff1f1f69d3f40858cb1ff05722da6f2 P ffedcedd84804834b6a74572c0b94cc8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 16:42:15.368921 23795 consensus_queue.cc:260] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 56c85d4ee801431f8c556a5dea08cf45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } }
I20260926 16:42:15.369171 23795 ts_tablet_manager.cc:1428] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 56c85d4ee801431f8c556a5dea08cf45: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 16:42:15.455534 23801 raft_consensus.cc:482] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 56c85d4ee801431f8c556a5dea08cf45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:42:15.455686 23801 raft_consensus.cc:504] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 56c85d4ee801431f8c556a5dea08cf45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } }
I20260926 16:42:15.456216 23801 leader_election.cc:288] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 56c85d4ee801431f8c556a5dea08cf45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a25832cf0b44dcbb487d80d238ed3f4 (127.21.12.65:39905), ffedcedd84804834b6a74572c0b94cc8 (127.21.12.67:36983)
I20260926 16:42:15.456759 23443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff1f1f69d3f40858cb1ff05722da6f2" candidate_uuid: "56c85d4ee801431f8c556a5dea08cf45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" is_pre_election: true
I20260926 16:42:15.456787 23724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff1f1f69d3f40858cb1ff05722da6f2" candidate_uuid: "56c85d4ee801431f8c556a5dea08cf45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffedcedd84804834b6a74572c0b94cc8" is_pre_election: true
I20260926 16:42:15.456941 23443 raft_consensus.cc:2454] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 1a25832cf0b44dcbb487d80d238ed3f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56c85d4ee801431f8c556a5dea08cf45 in term 0.
I20260926 16:42:15.456971 23724 raft_consensus.cc:2454] T 2ff1f1f69d3f40858cb1ff05722da6f2 P ffedcedd84804834b6a74572c0b94cc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56c85d4ee801431f8c556a5dea08cf45 in term 0.
I20260926 16:42:15.457309 23516 leader_election.cc:302] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 56c85d4ee801431f8c556a5dea08cf45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a25832cf0b44dcbb487d80d238ed3f4, 56c85d4ee801431f8c556a5dea08cf45; no voters:
I20260926 16:42:15.457679 23801 raft_consensus.cc:2781] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 56c85d4ee801431f8c556a5dea08cf45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:42:15.457739 23801 raft_consensus.cc:482] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 56c85d4ee801431f8c556a5dea08cf45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:42:15.457844 23801 raft_consensus.cc:3037] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 56c85d4ee801431f8c556a5dea08cf45 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:15.459148 23801 raft_consensus.cc:504] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 56c85d4ee801431f8c556a5dea08cf45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } }
I20260926 16:42:15.459513 23801 leader_election.cc:288] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 56c85d4ee801431f8c556a5dea08cf45 [CANDIDATE]: Term 1 election: Requested vote from peers 1a25832cf0b44dcbb487d80d238ed3f4 (127.21.12.65:39905), ffedcedd84804834b6a74572c0b94cc8 (127.21.12.67:36983)
I20260926 16:42:15.459944 23443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff1f1f69d3f40858cb1ff05722da6f2" candidate_uuid: "56c85d4ee801431f8c556a5dea08cf45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a25832cf0b44dcbb487d80d238ed3f4"
I20260926 16:42:15.459969 23724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff1f1f69d3f40858cb1ff05722da6f2" candidate_uuid: "56c85d4ee801431f8c556a5dea08cf45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffedcedd84804834b6a74572c0b94cc8"
I20260926 16:42:15.460079 23443 raft_consensus.cc:3037] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 1a25832cf0b44dcbb487d80d238ed3f4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:15.460122 23724 raft_consensus.cc:3037] T 2ff1f1f69d3f40858cb1ff05722da6f2 P ffedcedd84804834b6a74572c0b94cc8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:15.461264 23724 raft_consensus.cc:2454] T 2ff1f1f69d3f40858cb1ff05722da6f2 P ffedcedd84804834b6a74572c0b94cc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56c85d4ee801431f8c556a5dea08cf45 in term 1.
I20260926 16:42:15.461326 23443 raft_consensus.cc:2454] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 1a25832cf0b44dcbb487d80d238ed3f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56c85d4ee801431f8c556a5dea08cf45 in term 1.
I20260926 16:42:15.461583 23516 leader_election.cc:302] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 56c85d4ee801431f8c556a5dea08cf45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a25832cf0b44dcbb487d80d238ed3f4, 56c85d4ee801431f8c556a5dea08cf45; no voters:
I20260926 16:42:15.461824 23801 raft_consensus.cc:2781] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 56c85d4ee801431f8c556a5dea08cf45 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:15.461954 23801 raft_consensus.cc:686] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 56c85d4ee801431f8c556a5dea08cf45 [term 1 LEADER]: Becoming Leader. State: Replica: 56c85d4ee801431f8c556a5dea08cf45, State: Running, Role: LEADER
I20260926 16:42:15.462119 23801 consensus_queue.cc:237] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 56c85d4ee801431f8c556a5dea08cf45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } }
I20260926 16:42:15.463929 23286 catalog_manager.cc:5495] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 56c85d4ee801431f8c556a5dea08cf45 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56c85d4ee801431f8c556a5dea08cf45 (127.21.12.66). New cstate: current_term: 1 leader_uuid: "56c85d4ee801431f8c556a5dea08cf45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } health_report { overall_health: UNKNOWN } } }
I20260926 16:42:15.505147 23443 raft_consensus.cc:1261] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 1a25832cf0b44dcbb487d80d238ed3f4 [term 1 FOLLOWER]: Refusing update from remote peer 56c85d4ee801431f8c556a5dea08cf45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:42:15.505163 23724 raft_consensus.cc:1261] T 2ff1f1f69d3f40858cb1ff05722da6f2 P ffedcedd84804834b6a74572c0b94cc8 [term 1 FOLLOWER]: Refusing update from remote peer 56c85d4ee801431f8c556a5dea08cf45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:42:15.505576 23801 consensus_queue.cc:1035] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 56c85d4ee801431f8c556a5dea08cf45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:42:15.505767 23838 consensus_queue.cc:1035] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 56c85d4ee801431f8c556a5dea08cf45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 16:42:15.519959 23824 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 16:42:17.067423 23287 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:42:17.067566 23287 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:42:17.069833 23287 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 549ff0b7deb048668009a29152477376: Sending DeleteTablet for 3 replicas of tablet 2ff1f1f69d3f40858cb1ff05722da6f2
I20260926 16:42:17.070335 23564 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ff1f1f69d3f40858cb1ff05722da6f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:17 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:59650
I20260926 16:42:17.070374 23424 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ff1f1f69d3f40858cb1ff05722da6f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:17 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:34630
I20260926 16:42:17.070477 23704 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ff1f1f69d3f40858cb1ff05722da6f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:17 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:42052
I20260926 16:42:17.070744 23874 tablet_replica.cc:330] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 1a25832cf0b44dcbb487d80d238ed3f4: stopping tablet replica
I20260926 16:42:17.070771 23873 tablet_replica.cc:330] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 56c85d4ee801431f8c556a5dea08cf45: stopping tablet replica
I20260926 16:42:17.070904 23873 raft_consensus.cc:2229] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 56c85d4ee801431f8c556a5dea08cf45 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:42:17.070924 23874 raft_consensus.cc:2229] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 1a25832cf0b44dcbb487d80d238ed3f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:42:17.071048 23874 raft_consensus.cc:2258] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 1a25832cf0b44dcbb487d80d238ed3f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:42:17.071105 23873 raft_consensus.cc:2258] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 56c85d4ee801431f8c556a5dea08cf45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:42:17.071339 23875 tablet_replica.cc:330] T 2ff1f1f69d3f40858cb1ff05722da6f2 P ffedcedd84804834b6a74572c0b94cc8: stopping tablet replica
I20260926 16:42:17.071485 23875 raft_consensus.cc:2229] T 2ff1f1f69d3f40858cb1ff05722da6f2 P ffedcedd84804834b6a74572c0b94cc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:42:17.071642 23875 raft_consensus.cc:2258] T 2ff1f1f69d3f40858cb1ff05722da6f2 P ffedcedd84804834b6a74572c0b94cc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:42:17.071698 23874 ts_tablet_manager.cc:1905] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 1a25832cf0b44dcbb487d80d238ed3f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:42:17.072403 23875 ts_tablet_manager.cc:1905] T 2ff1f1f69d3f40858cb1ff05722da6f2 P ffedcedd84804834b6a74572c0b94cc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:42:17.073990 23874 ts_tablet_manager.cc:1918] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 1a25832cf0b44dcbb487d80d238ed3f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 16:42:17.074128 23874 log.cc:1196] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 1a25832cf0b44dcbb487d80d238ed3f4: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-0/wal/wals/2ff1f1f69d3f40858cb1ff05722da6f2
I20260926 16:42:17.074388 23873 ts_tablet_manager.cc:1905] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 56c85d4ee801431f8c556a5dea08cf45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:42:17.074488 23874 ts_tablet_manager.cc:1939] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 1a25832cf0b44dcbb487d80d238ed3f4: Deleting consensus metadata
I20260926 16:42:17.075184 23272 catalog_manager.cc:4841] TS 1a25832cf0b44dcbb487d80d238ed3f4 (127.21.12.65:39905): tablet 2ff1f1f69d3f40858cb1ff05722da6f2 (table test-table [id=283e783aac8c4aa9b029e286430bfb3d]) successfully deleted
I20260926 16:42:17.075397 23875 ts_tablet_manager.cc:1918] T 2ff1f1f69d3f40858cb1ff05722da6f2 P ffedcedd84804834b6a74572c0b94cc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 16:42:17.075484 23875 log.cc:1196] T 2ff1f1f69d3f40858cb1ff05722da6f2 P ffedcedd84804834b6a74572c0b94cc8: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-2/wal/wals/2ff1f1f69d3f40858cb1ff05722da6f2
I20260926 16:42:17.075718 23875 ts_tablet_manager.cc:1939] T 2ff1f1f69d3f40858cb1ff05722da6f2 P ffedcedd84804834b6a74572c0b94cc8: Deleting consensus metadata
I20260926 16:42:17.076332 23272 catalog_manager.cc:4841] TS ffedcedd84804834b6a74572c0b94cc8 (127.21.12.67:36983): tablet 2ff1f1f69d3f40858cb1ff05722da6f2 (table test-table [id=283e783aac8c4aa9b029e286430bfb3d]) successfully deleted
I20260926 16:42:17.076905 23873 ts_tablet_manager.cc:1918] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 56c85d4ee801431f8c556a5dea08cf45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 16:42:17.076968 23873 log.cc:1196] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 56c85d4ee801431f8c556a5dea08cf45: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-1/wal/wals/2ff1f1f69d3f40858cb1ff05722da6f2
I20260926 16:42:17.077234 23873 ts_tablet_manager.cc:1939] T 2ff1f1f69d3f40858cb1ff05722da6f2 P 56c85d4ee801431f8c556a5dea08cf45: Deleting consensus metadata
I20260926 16:42:17.077812 23273 catalog_manager.cc:4841] TS 56c85d4ee801431f8c556a5dea08cf45 (127.21.12.66:40961): tablet 2ff1f1f69d3f40858cb1ff05722da6f2 (table test-table [id=283e783aac8c4aa9b029e286430bfb3d]) successfully deleted
Sep 26 16:42:17 dist-test-slave-n1hw krb5kdc[23242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440937, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 16:42:17 dist-test-slave-n1hw krb5kdc[23242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440937, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:42:17.100870 23883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:17.091923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56610 (local address 127.21.12.126:33641)
0926 16:42:17.092186 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:17.092194 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:17.092298 (+ 104us) server_negotiation.cc:407] Connection header received
0926 16:42:17.092461 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:17.092464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:17.092515 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:17.092609 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:17.093300 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:17.093921 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:17.094756 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:17.095045 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:17.095154 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:17.097587 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:17.097591 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:17.097594 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:17.097618 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:17.099143 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:17.099561 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:17.099566 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:17.099568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:17.099653 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:17.099949 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:17.099954 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:17.099955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:17.100135 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:17.100281 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:17.100514 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 16:42:17.100640 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":81,"threads_started":1}
I20260926 16:42:17.103430 23287 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:42:17.103855 23287 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:42:17.108390 23424 tablet_service.cc:1467] Processing CreateTablet for tablet 9e9423e00ce443e28ee595ba4bf8cb4c (DEFAULT_TABLE table=test-table [id=da302a6aa682424188ba624ccef5dd60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:17.108523 23564 tablet_service.cc:1467] Processing CreateTablet for tablet 9e9423e00ce443e28ee595ba4bf8cb4c (DEFAULT_TABLE table=test-table [id=da302a6aa682424188ba624ccef5dd60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:17.108523 23704 tablet_service.cc:1467] Processing CreateTablet for tablet 9e9423e00ce443e28ee595ba4bf8cb4c (DEFAULT_TABLE table=test-table [id=da302a6aa682424188ba624ccef5dd60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:17.108700 23424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e9423e00ce443e28ee595ba4bf8cb4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:17.108777 23564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e9423e00ce443e28ee595ba4bf8cb4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:17.108783 23704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e9423e00ce443e28ee595ba4bf8cb4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:17.110718 23884 tablet_bootstrap.cc:492] T 9e9423e00ce443e28ee595ba4bf8cb4c P 1a25832cf0b44dcbb487d80d238ed3f4: Bootstrap starting.
I20260926 16:42:17.111244 23885 tablet_bootstrap.cc:492] T 9e9423e00ce443e28ee595ba4bf8cb4c P ffedcedd84804834b6a74572c0b94cc8: Bootstrap starting.
I20260926 16:42:17.111661 23884 tablet_bootstrap.cc:654] T 9e9423e00ce443e28ee595ba4bf8cb4c P 1a25832cf0b44dcbb487d80d238ed3f4: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:17.111701 23886 tablet_bootstrap.cc:492] T 9e9423e00ce443e28ee595ba4bf8cb4c P 56c85d4ee801431f8c556a5dea08cf45: Bootstrap starting.
I20260926 16:42:17.112161 23885 tablet_bootstrap.cc:654] T 9e9423e00ce443e28ee595ba4bf8cb4c P ffedcedd84804834b6a74572c0b94cc8: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:17.112526 23886 tablet_bootstrap.cc:654] T 9e9423e00ce443e28ee595ba4bf8cb4c P 56c85d4ee801431f8c556a5dea08cf45: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:17.112926 23884 tablet_bootstrap.cc:492] T 9e9423e00ce443e28ee595ba4bf8cb4c P 1a25832cf0b44dcbb487d80d238ed3f4: No bootstrap required, opened a new log
I20260926 16:42:17.113054 23884 ts_tablet_manager.cc:1397] T 9e9423e00ce443e28ee595ba4bf8cb4c P 1a25832cf0b44dcbb487d80d238ed3f4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 16:42:17.113281 23885 tablet_bootstrap.cc:492] T 9e9423e00ce443e28ee595ba4bf8cb4c P ffedcedd84804834b6a74572c0b94cc8: No bootstrap required, opened a new log
I20260926 16:42:17.113400 23885 ts_tablet_manager.cc:1397] T 9e9423e00ce443e28ee595ba4bf8cb4c P ffedcedd84804834b6a74572c0b94cc8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 16:42:17.113739 23884 raft_consensus.cc:348] T 9e9423e00ce443e28ee595ba4bf8cb4c P 1a25832cf0b44dcbb487d80d238ed3f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } }
I20260926 16:42:17.113919 23884 raft_consensus.cc:374] T 9e9423e00ce443e28ee595ba4bf8cb4c P 1a25832cf0b44dcbb487d80d238ed3f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:17.113914 23885 raft_consensus.cc:348] T 9e9423e00ce443e28ee595ba4bf8cb4c P ffedcedd84804834b6a74572c0b94cc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } }
I20260926 16:42:17.113998 23884 raft_consensus.cc:729] T 9e9423e00ce443e28ee595ba4bf8cb4c P 1a25832cf0b44dcbb487d80d238ed3f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a25832cf0b44dcbb487d80d238ed3f4, State: Initialized, Role: FOLLOWER
I20260926 16:42:17.114038 23885 raft_consensus.cc:374] T 9e9423e00ce443e28ee595ba4bf8cb4c P ffedcedd84804834b6a74572c0b94cc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:17.114082 23885 raft_consensus.cc:729] T 9e9423e00ce443e28ee595ba4bf8cb4c P ffedcedd84804834b6a74572c0b94cc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffedcedd84804834b6a74572c0b94cc8, State: Initialized, Role: FOLLOWER
I20260926 16:42:17.114130 23884 consensus_queue.cc:260] T 9e9423e00ce443e28ee595ba4bf8cb4c P 1a25832cf0b44dcbb487d80d238ed3f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } }
I20260926 16:42:17.114220 23885 consensus_queue.cc:260] T 9e9423e00ce443e28ee595ba4bf8cb4c P ffedcedd84804834b6a74572c0b94cc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } }
I20260926 16:42:17.114454 23886 tablet_bootstrap.cc:492] T 9e9423e00ce443e28ee595ba4bf8cb4c P 56c85d4ee801431f8c556a5dea08cf45: No bootstrap required, opened a new log
I20260926 16:42:17.114490 23885 ts_tablet_manager.cc:1428] T 9e9423e00ce443e28ee595ba4bf8cb4c P ffedcedd84804834b6a74572c0b94cc8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 16:42:17.114514 23884 ts_tablet_manager.cc:1428] T 9e9423e00ce443e28ee595ba4bf8cb4c P 1a25832cf0b44dcbb487d80d238ed3f4: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260926 16:42:17.114526 23886 ts_tablet_manager.cc:1397] T 9e9423e00ce443e28ee595ba4bf8cb4c P 56c85d4ee801431f8c556a5dea08cf45: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 16:42:17.114904 23886 raft_consensus.cc:348] T 9e9423e00ce443e28ee595ba4bf8cb4c P 56c85d4ee801431f8c556a5dea08cf45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } }
I20260926 16:42:17.115010 23886 raft_consensus.cc:374] T 9e9423e00ce443e28ee595ba4bf8cb4c P 56c85d4ee801431f8c556a5dea08cf45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:17.115048 23886 raft_consensus.cc:729] T 9e9423e00ce443e28ee595ba4bf8cb4c P 56c85d4ee801431f8c556a5dea08cf45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56c85d4ee801431f8c556a5dea08cf45, State: Initialized, Role: FOLLOWER
I20260926 16:42:17.115172 23886 consensus_queue.cc:260] T 9e9423e00ce443e28ee595ba4bf8cb4c P 56c85d4ee801431f8c556a5dea08cf45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } }
I20260926 16:42:17.115464 23886 ts_tablet_manager.cc:1428] T 9e9423e00ce443e28ee595ba4bf8cb4c P 56c85d4ee801431f8c556a5dea08cf45: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 16:42:17.180876 23800 raft_consensus.cc:482] T 9e9423e00ce443e28ee595ba4bf8cb4c P ffedcedd84804834b6a74572c0b94cc8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:42:17.181006 23800 raft_consensus.cc:504] T 9e9423e00ce443e28ee595ba4bf8cb4c P ffedcedd84804834b6a74572c0b94cc8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } }
I20260926 16:42:17.181411 23800 leader_election.cc:288] T 9e9423e00ce443e28ee595ba4bf8cb4c P ffedcedd84804834b6a74572c0b94cc8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a25832cf0b44dcbb487d80d238ed3f4 (127.21.12.65:39905), 56c85d4ee801431f8c556a5dea08cf45 (127.21.12.66:40961)
I20260926 16:42:17.181839 23443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e9423e00ce443e28ee595ba4bf8cb4c" candidate_uuid: "ffedcedd84804834b6a74572c0b94cc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" is_pre_election: true
I20260926 16:42:17.181913 23584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e9423e00ce443e28ee595ba4bf8cb4c" candidate_uuid: "ffedcedd84804834b6a74572c0b94cc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56c85d4ee801431f8c556a5dea08cf45" is_pre_election: true
I20260926 16:42:17.181998 23443 raft_consensus.cc:2454] T 9e9423e00ce443e28ee595ba4bf8cb4c P 1a25832cf0b44dcbb487d80d238ed3f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffedcedd84804834b6a74572c0b94cc8 in term 0.
I20260926 16:42:17.182106 23584 raft_consensus.cc:2454] T 9e9423e00ce443e28ee595ba4bf8cb4c P 56c85d4ee801431f8c556a5dea08cf45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffedcedd84804834b6a74572c0b94cc8 in term 0.
I20260926 16:42:17.182394 23656 leader_election.cc:302] T 9e9423e00ce443e28ee595ba4bf8cb4c P ffedcedd84804834b6a74572c0b94cc8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a25832cf0b44dcbb487d80d238ed3f4, ffedcedd84804834b6a74572c0b94cc8; no voters:
I20260926 16:42:17.182621 23800 raft_consensus.cc:2781] T 9e9423e00ce443e28ee595ba4bf8cb4c P ffedcedd84804834b6a74572c0b94cc8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:42:17.182682 23800 raft_consensus.cc:482] T 9e9423e00ce443e28ee595ba4bf8cb4c P ffedcedd84804834b6a74572c0b94cc8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:42:17.182722 23800 raft_consensus.cc:3037] T 9e9423e00ce443e28ee595ba4bf8cb4c P ffedcedd84804834b6a74572c0b94cc8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:17.183646 23800 raft_consensus.cc:504] T 9e9423e00ce443e28ee595ba4bf8cb4c P ffedcedd84804834b6a74572c0b94cc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } }
I20260926 16:42:17.183974 23800 leader_election.cc:288] T 9e9423e00ce443e28ee595ba4bf8cb4c P ffedcedd84804834b6a74572c0b94cc8 [CANDIDATE]: Term 1 election: Requested vote from peers 1a25832cf0b44dcbb487d80d238ed3f4 (127.21.12.65:39905), 56c85d4ee801431f8c556a5dea08cf45 (127.21.12.66:40961)
I20260926 16:42:17.184306 23443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e9423e00ce443e28ee595ba4bf8cb4c" candidate_uuid: "ffedcedd84804834b6a74572c0b94cc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a25832cf0b44dcbb487d80d238ed3f4"
I20260926 16:42:17.184326 23584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e9423e00ce443e28ee595ba4bf8cb4c" candidate_uuid: "ffedcedd84804834b6a74572c0b94cc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56c85d4ee801431f8c556a5dea08cf45"
I20260926 16:42:17.184433 23443 raft_consensus.cc:3037] T 9e9423e00ce443e28ee595ba4bf8cb4c P 1a25832cf0b44dcbb487d80d238ed3f4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:17.184433 23584 raft_consensus.cc:3037] T 9e9423e00ce443e28ee595ba4bf8cb4c P 56c85d4ee801431f8c556a5dea08cf45 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:17.185525 23584 raft_consensus.cc:2454] T 9e9423e00ce443e28ee595ba4bf8cb4c P 56c85d4ee801431f8c556a5dea08cf45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffedcedd84804834b6a74572c0b94cc8 in term 1.
I20260926 16:42:17.185525 23443 raft_consensus.cc:2454] T 9e9423e00ce443e28ee595ba4bf8cb4c P 1a25832cf0b44dcbb487d80d238ed3f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffedcedd84804834b6a74572c0b94cc8 in term 1.
I20260926 16:42:17.185837 23656 leader_election.cc:302] T 9e9423e00ce443e28ee595ba4bf8cb4c P ffedcedd84804834b6a74572c0b94cc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1a25832cf0b44dcbb487d80d238ed3f4, 56c85d4ee801431f8c556a5dea08cf45, ffedcedd84804834b6a74572c0b94cc8; no voters:
I20260926 16:42:17.186029 23800 raft_consensus.cc:2781] T 9e9423e00ce443e28ee595ba4bf8cb4c P ffedcedd84804834b6a74572c0b94cc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:17.186115 23800 raft_consensus.cc:686] T 9e9423e00ce443e28ee595ba4bf8cb4c P ffedcedd84804834b6a74572c0b94cc8 [term 1 LEADER]: Becoming Leader. State: Replica: ffedcedd84804834b6a74572c0b94cc8, State: Running, Role: LEADER
I20260926 16:42:17.186264 23800 consensus_queue.cc:237] T 9e9423e00ce443e28ee595ba4bf8cb4c P ffedcedd84804834b6a74572c0b94cc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } }
I20260926 16:42:17.187923 23287 catalog_manager.cc:5495] T 9e9423e00ce443e28ee595ba4bf8cb4c P ffedcedd84804834b6a74572c0b94cc8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ffedcedd84804834b6a74572c0b94cc8 (127.21.12.67). New cstate: current_term: 1 leader_uuid: "ffedcedd84804834b6a74572c0b94cc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } health_report { overall_health: HEALTHY } } }
I20260926 16:42:17.218336 23893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:17.215379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42076 (local address 127.21.12.67:36983)
0926 16:42:17.215601 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:17.215607 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:17.215619 (+ 12us) server_negotiation.cc:407] Connection header received
0926 16:42:17.215654 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:17.215656 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:17.215722 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:17.215826 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:42:17.216085 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:17.216580 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:17.217463 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:17.217674 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:17.217787 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:17.217991 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:42:17.218053 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:42:17.218109 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:17.218186 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 16:42:17.218226 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":98,"threads_started":1}
I20260926 16:42:17.226934 23896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:17.223767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.67:36983 (local address 127.21.12.67:35645)
0926 16:42:17.224043 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:17.224055 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:17.224142 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:17.224408 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:17.224410 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:17.224417 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 16:42:17.224497 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:17.224501 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:17.225180 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:17.225182 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:17.225783 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:17.225791 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:17.226211 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:17.226214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:17.226345 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:17.226374 (+ 29us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 16:42:17.226637 (+ 263us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 16:42:17.226644 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 16:42:17.226702 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 16:42:17.226810 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":27,"thread_start_us":129,"threads_started":1}
I20260926 16:42:17.226934 23893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:17.223882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:35645 (local address 127.21.12.67:36983)
0926 16:42:17.223997 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:17.224000 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:17.224156 (+ 156us) server_negotiation.cc:407] Connection header received
0926 16:42:17.224199 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:17.224206 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:17.224258 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:17.224346 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:42:17.224602 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:17.225079 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:17.225879 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:17.226093 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:17.226256 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:17.226492 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:42:17.226547 (+ 55us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:42:17.226621 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:17.226709 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 16:42:17.226794 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":46}
I20260926 16:42:17.232440 23584 raft_consensus.cc:1261] T 9e9423e00ce443e28ee595ba4bf8cb4c P 56c85d4ee801431f8c556a5dea08cf45 [term 1 FOLLOWER]: Refusing update from remote peer ffedcedd84804834b6a74572c0b94cc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:42:17.232474 23443 raft_consensus.cc:1261] T 9e9423e00ce443e28ee595ba4bf8cb4c P 1a25832cf0b44dcbb487d80d238ed3f4 [term 1 FOLLOWER]: Refusing update from remote peer ffedcedd84804834b6a74572c0b94cc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:42:17.232827 23800 consensus_queue.cc:1035] T 9e9423e00ce443e28ee595ba4bf8cb4c P ffedcedd84804834b6a74572c0b94cc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:42:17.232968 23854 consensus_queue.cc:1035] T 9e9423e00ce443e28ee595ba4bf8cb4c P ffedcedd84804834b6a74572c0b94cc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:42:17.263147 23287 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:42:17.263301 23287 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:42:17.265239 23287 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 549ff0b7deb048668009a29152477376: Sending DeleteTablet for 3 replicas of tablet 9e9423e00ce443e28ee595ba4bf8cb4c
I20260926 16:42:17.265811 23564 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e9423e00ce443e28ee595ba4bf8cb4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:17 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:59650
I20260926 16:42:17.265981 23873 tablet_replica.cc:330] T 9e9423e00ce443e28ee595ba4bf8cb4c P 56c85d4ee801431f8c556a5dea08cf45: stopping tablet replica
I20260926 16:42:17.265962 23424 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e9423e00ce443e28ee595ba4bf8cb4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:17 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:34630
I20260926 16:42:17.266103 23873 raft_consensus.cc:2229] T 9e9423e00ce443e28ee595ba4bf8cb4c P 56c85d4ee801431f8c556a5dea08cf45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:42:17.266215 23873 raft_consensus.cc:2258] T 9e9423e00ce443e28ee595ba4bf8cb4c P 56c85d4ee801431f8c556a5dea08cf45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:42:17.266374 23874 tablet_replica.cc:330] T 9e9423e00ce443e28ee595ba4bf8cb4c P 1a25832cf0b44dcbb487d80d238ed3f4: stopping tablet replica
I20260926 16:42:17.266479 23874 raft_consensus.cc:2229] T 9e9423e00ce443e28ee595ba4bf8cb4c P 1a25832cf0b44dcbb487d80d238ed3f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:42:17.266582 23874 raft_consensus.cc:2258] T 9e9423e00ce443e28ee595ba4bf8cb4c P 1a25832cf0b44dcbb487d80d238ed3f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:42:17.266928 23287 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:42:17.267307 23287 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:42:17.267345 23873 ts_tablet_manager.cc:1905] T 9e9423e00ce443e28ee595ba4bf8cb4c P 56c85d4ee801431f8c556a5dea08cf45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:42:17.268126 23703 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e9423e00ce443e28ee595ba4bf8cb4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:17 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:42052
I20260926 16:42:17.268291 23875 tablet_replica.cc:330] T 9e9423e00ce443e28ee595ba4bf8cb4c P ffedcedd84804834b6a74572c0b94cc8: stopping tablet replica
I20260926 16:42:17.268402 23875 raft_consensus.cc:2229] T 9e9423e00ce443e28ee595ba4bf8cb4c P ffedcedd84804834b6a74572c0b94cc8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:42:17.268667 23875 raft_consensus.cc:2258] T 9e9423e00ce443e28ee595ba4bf8cb4c P ffedcedd84804834b6a74572c0b94cc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:42:17.269348 23875 ts_tablet_manager.cc:1905] T 9e9423e00ce443e28ee595ba4bf8cb4c P ffedcedd84804834b6a74572c0b94cc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:42:17.270059 23873 ts_tablet_manager.cc:1918] T 9e9423e00ce443e28ee595ba4bf8cb4c P 56c85d4ee801431f8c556a5dea08cf45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:42:17.270134 23873 log.cc:1196] T 9e9423e00ce443e28ee595ba4bf8cb4c P 56c85d4ee801431f8c556a5dea08cf45: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-1/wal/wals/9e9423e00ce443e28ee595ba4bf8cb4c
I20260926 16:42:17.270402 23873 ts_tablet_manager.cc:1939] T 9e9423e00ce443e28ee595ba4bf8cb4c P 56c85d4ee801431f8c556a5dea08cf45: Deleting consensus metadata
I20260926 16:42:17.271696 23875 ts_tablet_manager.cc:1918] T 9e9423e00ce443e28ee595ba4bf8cb4c P ffedcedd84804834b6a74572c0b94cc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:42:17.271768 23875 log.cc:1196] T 9e9423e00ce443e28ee595ba4bf8cb4c P ffedcedd84804834b6a74572c0b94cc8: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-2/wal/wals/9e9423e00ce443e28ee595ba4bf8cb4c
I20260926 16:42:17.272006 23875 ts_tablet_manager.cc:1939] T 9e9423e00ce443e28ee595ba4bf8cb4c P ffedcedd84804834b6a74572c0b94cc8: Deleting consensus metadata
I20260926 16:42:17.272459 23424 tablet_service.cc:1467] Processing CreateTablet for tablet 9a904f37d95847eb823003a385767e93 (DEFAULT_TABLE table=test-table [id=9315a0ba0d0a49bb942bc4f6996ba58f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:17.272616 23272 catalog_manager.cc:4841] TS ffedcedd84804834b6a74572c0b94cc8 (127.21.12.67:36983): tablet 9e9423e00ce443e28ee595ba4bf8cb4c (table test-table [id=da302a6aa682424188ba624ccef5dd60]) successfully deleted
I20260926 16:42:17.272717 23424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a904f37d95847eb823003a385767e93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:17.272851 23564 tablet_service.cc:1467] Processing CreateTablet for tablet 9a904f37d95847eb823003a385767e93 (DEFAULT_TABLE table=test-table [id=9315a0ba0d0a49bb942bc4f6996ba58f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:17.273085 23564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a904f37d95847eb823003a385767e93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:17.273880 23273 catalog_manager.cc:4841] TS 56c85d4ee801431f8c556a5dea08cf45 (127.21.12.66:40961): tablet 9e9423e00ce443e28ee595ba4bf8cb4c (table test-table [id=da302a6aa682424188ba624ccef5dd60]) successfully deleted
I20260926 16:42:17.274181 23703 tablet_service.cc:1467] Processing CreateTablet for tablet 9a904f37d95847eb823003a385767e93 (DEFAULT_TABLE table=test-table [id=9315a0ba0d0a49bb942bc4f6996ba58f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:17.274415 23703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a904f37d95847eb823003a385767e93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:17.275122 23886 tablet_bootstrap.cc:492] T 9a904f37d95847eb823003a385767e93 P 56c85d4ee801431f8c556a5dea08cf45: Bootstrap starting.
I20260926 16:42:17.275862 23886 tablet_bootstrap.cc:654] T 9a904f37d95847eb823003a385767e93 P 56c85d4ee801431f8c556a5dea08cf45: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:17.276568 23885 tablet_bootstrap.cc:492] T 9a904f37d95847eb823003a385767e93 P ffedcedd84804834b6a74572c0b94cc8: Bootstrap starting.
I20260926 16:42:17.276598 23884 tablet_bootstrap.cc:492] T 9a904f37d95847eb823003a385767e93 P 1a25832cf0b44dcbb487d80d238ed3f4: Bootstrap starting.
I20260926 16:42:17.277207 23874 ts_tablet_manager.cc:1905] T 9e9423e00ce443e28ee595ba4bf8cb4c P 1a25832cf0b44dcbb487d80d238ed3f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:42:17.277354 23885 tablet_bootstrap.cc:654] T 9a904f37d95847eb823003a385767e93 P ffedcedd84804834b6a74572c0b94cc8: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:17.277540 23884 tablet_bootstrap.cc:654] T 9a904f37d95847eb823003a385767e93 P 1a25832cf0b44dcbb487d80d238ed3f4: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:17.278590 23886 tablet_bootstrap.cc:492] T 9a904f37d95847eb823003a385767e93 P 56c85d4ee801431f8c556a5dea08cf45: No bootstrap required, opened a new log
I20260926 16:42:17.278676 23886 ts_tablet_manager.cc:1397] T 9a904f37d95847eb823003a385767e93 P 56c85d4ee801431f8c556a5dea08cf45: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 16:42:17.278708 23884 tablet_bootstrap.cc:492] T 9a904f37d95847eb823003a385767e93 P 1a25832cf0b44dcbb487d80d238ed3f4: No bootstrap required, opened a new log
I20260926 16:42:17.278777 23884 ts_tablet_manager.cc:1397] T 9a904f37d95847eb823003a385767e93 P 1a25832cf0b44dcbb487d80d238ed3f4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 16:42:17.279042 23886 raft_consensus.cc:348] T 9a904f37d95847eb823003a385767e93 P 56c85d4ee801431f8c556a5dea08cf45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } }
I20260926 16:42:17.279163 23886 raft_consensus.cc:374] T 9a904f37d95847eb823003a385767e93 P 56c85d4ee801431f8c556a5dea08cf45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:17.279197 23886 raft_consensus.cc:729] T 9a904f37d95847eb823003a385767e93 P 56c85d4ee801431f8c556a5dea08cf45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56c85d4ee801431f8c556a5dea08cf45, State: Initialized, Role: FOLLOWER
I20260926 16:42:17.279181 23884 raft_consensus.cc:348] T 9a904f37d95847eb823003a385767e93 P 1a25832cf0b44dcbb487d80d238ed3f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } }
I20260926 16:42:17.279280 23884 raft_consensus.cc:374] T 9a904f37d95847eb823003a385767e93 P 1a25832cf0b44dcbb487d80d238ed3f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:17.279312 23884 raft_consensus.cc:729] T 9a904f37d95847eb823003a385767e93 P 1a25832cf0b44dcbb487d80d238ed3f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a25832cf0b44dcbb487d80d238ed3f4, State: Initialized, Role: FOLLOWER
I20260926 16:42:17.279335 23886 consensus_queue.cc:260] T 9a904f37d95847eb823003a385767e93 P 56c85d4ee801431f8c556a5dea08cf45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } }
I20260926 16:42:17.279420 23884 consensus_queue.cc:260] T 9a904f37d95847eb823003a385767e93 P 1a25832cf0b44dcbb487d80d238ed3f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } }
I20260926 16:42:17.279507 23874 ts_tablet_manager.cc:1918] T 9e9423e00ce443e28ee595ba4bf8cb4c P 1a25832cf0b44dcbb487d80d238ed3f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:42:17.279569 23874 log.cc:1196] T 9e9423e00ce443e28ee595ba4bf8cb4c P 1a25832cf0b44dcbb487d80d238ed3f4: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-0/wal/wals/9e9423e00ce443e28ee595ba4bf8cb4c
I20260926 16:42:17.279650 23886 ts_tablet_manager.cc:1428] T 9a904f37d95847eb823003a385767e93 P 56c85d4ee801431f8c556a5dea08cf45: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 16:42:17.279663 23884 ts_tablet_manager.cc:1428] T 9a904f37d95847eb823003a385767e93 P 1a25832cf0b44dcbb487d80d238ed3f4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 16:42:17.279843 23874 ts_tablet_manager.cc:1939] T 9e9423e00ce443e28ee595ba4bf8cb4c P 1a25832cf0b44dcbb487d80d238ed3f4: Deleting consensus metadata
I20260926 16:42:17.280740 23885 tablet_bootstrap.cc:492] T 9a904f37d95847eb823003a385767e93 P ffedcedd84804834b6a74572c0b94cc8: No bootstrap required, opened a new log
I20260926 16:42:17.280821 23885 ts_tablet_manager.cc:1397] T 9a904f37d95847eb823003a385767e93 P ffedcedd84804834b6a74572c0b94cc8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 16:42:17.281212 23885 raft_consensus.cc:348] T 9a904f37d95847eb823003a385767e93 P ffedcedd84804834b6a74572c0b94cc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } }
I20260926 16:42:17.281324 23885 raft_consensus.cc:374] T 9a904f37d95847eb823003a385767e93 P ffedcedd84804834b6a74572c0b94cc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:17.281313 23272 catalog_manager.cc:4841] TS 1a25832cf0b44dcbb487d80d238ed3f4 (127.21.12.65:39905): tablet 9e9423e00ce443e28ee595ba4bf8cb4c (table test-table [id=da302a6aa682424188ba624ccef5dd60]) successfully deleted
I20260926 16:42:17.281356 23885 raft_consensus.cc:729] T 9a904f37d95847eb823003a385767e93 P ffedcedd84804834b6a74572c0b94cc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffedcedd84804834b6a74572c0b94cc8, State: Initialized, Role: FOLLOWER
I20260926 16:42:17.281469 23885 consensus_queue.cc:260] T 9a904f37d95847eb823003a385767e93 P ffedcedd84804834b6a74572c0b94cc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } }
I20260926 16:42:17.281721 23885 ts_tablet_manager.cc:1428] T 9a904f37d95847eb823003a385767e93 P ffedcedd84804834b6a74572c0b94cc8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 16:42:17.373499 23859 raft_consensus.cc:482] T 9a904f37d95847eb823003a385767e93 P ffedcedd84804834b6a74572c0b94cc8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:42:17.373647 23859 raft_consensus.cc:504] T 9a904f37d95847eb823003a385767e93 P ffedcedd84804834b6a74572c0b94cc8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } }
I20260926 16:42:17.374220 23859 leader_election.cc:288] T 9a904f37d95847eb823003a385767e93 P ffedcedd84804834b6a74572c0b94cc8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a25832cf0b44dcbb487d80d238ed3f4 (127.21.12.65:39905), 56c85d4ee801431f8c556a5dea08cf45 (127.21.12.66:40961)
I20260926 16:42:17.374532 23443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a904f37d95847eb823003a385767e93" candidate_uuid: "ffedcedd84804834b6a74572c0b94cc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" is_pre_election: true
I20260926 16:42:17.374624 23583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a904f37d95847eb823003a385767e93" candidate_uuid: "ffedcedd84804834b6a74572c0b94cc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56c85d4ee801431f8c556a5dea08cf45" is_pre_election: true
I20260926 16:42:17.374722 23443 raft_consensus.cc:2454] T 9a904f37d95847eb823003a385767e93 P 1a25832cf0b44dcbb487d80d238ed3f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffedcedd84804834b6a74572c0b94cc8 in term 0.
I20260926 16:42:17.374743 23583 raft_consensus.cc:2454] T 9a904f37d95847eb823003a385767e93 P 56c85d4ee801431f8c556a5dea08cf45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffedcedd84804834b6a74572c0b94cc8 in term 0.
I20260926 16:42:17.375069 23657 leader_election.cc:302] T 9a904f37d95847eb823003a385767e93 P ffedcedd84804834b6a74572c0b94cc8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56c85d4ee801431f8c556a5dea08cf45, ffedcedd84804834b6a74572c0b94cc8; no voters:
I20260926 16:42:17.375232 23859 raft_consensus.cc:2781] T 9a904f37d95847eb823003a385767e93 P ffedcedd84804834b6a74572c0b94cc8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:42:17.375289 23859 raft_consensus.cc:482] T 9a904f37d95847eb823003a385767e93 P ffedcedd84804834b6a74572c0b94cc8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:42:17.375336 23859 raft_consensus.cc:3037] T 9a904f37d95847eb823003a385767e93 P ffedcedd84804834b6a74572c0b94cc8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:17.376433 23859 raft_consensus.cc:504] T 9a904f37d95847eb823003a385767e93 P ffedcedd84804834b6a74572c0b94cc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } }
I20260926 16:42:17.376820 23859 leader_election.cc:288] T 9a904f37d95847eb823003a385767e93 P ffedcedd84804834b6a74572c0b94cc8 [CANDIDATE]: Term 1 election: Requested vote from peers 1a25832cf0b44dcbb487d80d238ed3f4 (127.21.12.65:39905), 56c85d4ee801431f8c556a5dea08cf45 (127.21.12.66:40961)
I20260926 16:42:17.377175 23443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a904f37d95847eb823003a385767e93" candidate_uuid: "ffedcedd84804834b6a74572c0b94cc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a25832cf0b44dcbb487d80d238ed3f4"
I20260926 16:42:17.377203 23583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a904f37d95847eb823003a385767e93" candidate_uuid: "ffedcedd84804834b6a74572c0b94cc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56c85d4ee801431f8c556a5dea08cf45"
I20260926 16:42:17.377302 23583 raft_consensus.cc:3037] T 9a904f37d95847eb823003a385767e93 P 56c85d4ee801431f8c556a5dea08cf45 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:17.377303 23443 raft_consensus.cc:3037] T 9a904f37d95847eb823003a385767e93 P 1a25832cf0b44dcbb487d80d238ed3f4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:17.378504 23583 raft_consensus.cc:2454] T 9a904f37d95847eb823003a385767e93 P 56c85d4ee801431f8c556a5dea08cf45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffedcedd84804834b6a74572c0b94cc8 in term 1.
I20260926 16:42:17.378830 23657 leader_election.cc:302] T 9a904f37d95847eb823003a385767e93 P ffedcedd84804834b6a74572c0b94cc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56c85d4ee801431f8c556a5dea08cf45, ffedcedd84804834b6a74572c0b94cc8; no voters:
I20260926 16:42:17.378878 23443 raft_consensus.cc:2454] T 9a904f37d95847eb823003a385767e93 P 1a25832cf0b44dcbb487d80d238ed3f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffedcedd84804834b6a74572c0b94cc8 in term 1.
I20260926 16:42:17.378983 23859 raft_consensus.cc:2781] T 9a904f37d95847eb823003a385767e93 P ffedcedd84804834b6a74572c0b94cc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:17.379089 23859 raft_consensus.cc:686] T 9a904f37d95847eb823003a385767e93 P ffedcedd84804834b6a74572c0b94cc8 [term 1 LEADER]: Becoming Leader. State: Replica: ffedcedd84804834b6a74572c0b94cc8, State: Running, Role: LEADER
I20260926 16:42:17.379285 23859 consensus_queue.cc:237] T 9a904f37d95847eb823003a385767e93 P ffedcedd84804834b6a74572c0b94cc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } }
I20260926 16:42:17.381044 23287 catalog_manager.cc:5495] T 9a904f37d95847eb823003a385767e93 P ffedcedd84804834b6a74572c0b94cc8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ffedcedd84804834b6a74572c0b94cc8 (127.21.12.67). New cstate: current_term: 1 leader_uuid: "ffedcedd84804834b6a74572c0b94cc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ffedcedd84804834b6a74572c0b94cc8" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36983 } health_report { overall_health: HEALTHY } } }
I20260926 16:42:17.419535 23443 raft_consensus.cc:1261] T 9a904f37d95847eb823003a385767e93 P 1a25832cf0b44dcbb487d80d238ed3f4 [term 1 FOLLOWER]: Refusing update from remote peer ffedcedd84804834b6a74572c0b94cc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:42:17.419535 23583 raft_consensus.cc:1261] T 9a904f37d95847eb823003a385767e93 P 56c85d4ee801431f8c556a5dea08cf45 [term 1 FOLLOWER]: Refusing update from remote peer ffedcedd84804834b6a74572c0b94cc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:42:17.419970 23800 consensus_queue.cc:1035] T 9a904f37d95847eb823003a385767e93 P ffedcedd84804834b6a74572c0b94cc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a25832cf0b44dcbb487d80d238ed3f4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:42:17.420136 23859 consensus_queue.cc:1035] T 9a904f37d95847eb823003a385767e93 P ffedcedd84804834b6a74572c0b94cc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56c85d4ee801431f8c556a5dea08cf45" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 40961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 16:42:17.434361 23852 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 16:42:18.983351 23287 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:42:18.983496 23287 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:42:18.985780 23287 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 549ff0b7deb048668009a29152477376: Sending DeleteTablet for 3 replicas of tablet 9a904f37d95847eb823003a385767e93
I20260926 16:42:18.986336 23424 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a904f37d95847eb823003a385767e93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:18 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:34630
I20260926 16:42:18.986450 23703 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a904f37d95847eb823003a385767e93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:18 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:42052
I20260926 16:42:18.986462 23564 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a904f37d95847eb823003a385767e93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:18 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:59650
I20260926 16:42:18.986754 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 23359
I20260926 16:42:18.986804 23925 tablet_replica.cc:330] T 9a904f37d95847eb823003a385767e93 P 1a25832cf0b44dcbb487d80d238ed3f4: stopping tablet replica
I20260926 16:42:18.988044 23927 tablet_replica.cc:330] T 9a904f37d95847eb823003a385767e93 P ffedcedd84804834b6a74572c0b94cc8: stopping tablet replica
I20260926 16:42:18.988234 23926 tablet_replica.cc:330] T 9a904f37d95847eb823003a385767e93 P 56c85d4ee801431f8c556a5dea08cf45: stopping tablet replica
I20260926 16:42:18.988379 23926 raft_consensus.cc:2229] T 9a904f37d95847eb823003a385767e93 P 56c85d4ee801431f8c556a5dea08cf45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:42:18.988534 23926 raft_consensus.cc:2258] T 9a904f37d95847eb823003a385767e93 P 56c85d4ee801431f8c556a5dea08cf45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:42:18.988776 23927 raft_consensus.cc:2229] T 9a904f37d95847eb823003a385767e93 P ffedcedd84804834b6a74572c0b94cc8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:42:18.989050 23927 raft_consensus.cc:2258] T 9a904f37d95847eb823003a385767e93 P ffedcedd84804834b6a74572c0b94cc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:42:18.989282 23926 ts_tablet_manager.cc:1905] T 9a904f37d95847eb823003a385767e93 P 56c85d4ee801431f8c556a5dea08cf45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:42:18.989650 23927 ts_tablet_manager.cc:1905] T 9a904f37d95847eb823003a385767e93 P ffedcedd84804834b6a74572c0b94cc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:42:18.991914 23927 ts_tablet_manager.cc:1918] T 9a904f37d95847eb823003a385767e93 P ffedcedd84804834b6a74572c0b94cc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 16:42:18.992017 23927 log.cc:1196] T 9a904f37d95847eb823003a385767e93 P ffedcedd84804834b6a74572c0b94cc8: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-2/wal/wals/9a904f37d95847eb823003a385767e93
I20260926 16:42:18.992287 23927 ts_tablet_manager.cc:1939] T 9a904f37d95847eb823003a385767e93 P ffedcedd84804834b6a74572c0b94cc8: Deleting consensus metadata
I20260926 16:42:18.992759 23926 ts_tablet_manager.cc:1918] T 9a904f37d95847eb823003a385767e93 P 56c85d4ee801431f8c556a5dea08cf45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 16:42:18.992848 23926 log.cc:1196] T 9a904f37d95847eb823003a385767e93 P 56c85d4ee801431f8c556a5dea08cf45: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440924937351-21553-0/minicluster-data/ts-1/wal/wals/9a904f37d95847eb823003a385767e93
I20260926 16:42:18.992969 23272 catalog_manager.cc:4841] TS ffedcedd84804834b6a74572c0b94cc8 (127.21.12.67:36983): tablet 9a904f37d95847eb823003a385767e93 (table test-table [id=9315a0ba0d0a49bb942bc4f6996ba58f]) successfully deleted
I20260926 16:42:18.993103 23926 ts_tablet_manager.cc:1939] T 9a904f37d95847eb823003a385767e93 P 56c85d4ee801431f8c556a5dea08cf45: Deleting consensus metadata
I20260926 16:42:18.994022 23273 catalog_manager.cc:4841] TS 56c85d4ee801431f8c556a5dea08cf45 (127.21.12.66:40961): tablet 9a904f37d95847eb823003a385767e93 (table test-table [id=9315a0ba0d0a49bb942bc4f6996ba58f]) successfully deleted
W20260926 16:42:18.994963 23272 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.12.65:39905 (error 108)
I20260926 16:42:18.995113 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 23499
I20260926 16:42:18.996691 23929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:18.996234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:39905 (local address 127.0.0.1:34644)
0926 16:42:18.996544 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:18.996608 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.65:39905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":85,"threads_started":1}
W20260926 16:42:18.996829 23272 catalog_manager.cc:4584] TS 1a25832cf0b44dcbb487d80d238ed3f4 (127.21.12.65:39905): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9a904f37d95847eb823003a385767e93: Network error: Client connection negotiation failed: client connection to 127.21.12.65:39905: connect: Connection refused (error 111)
I20260926 16:42:19.002524 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 23639
I20260926 16:42:19.009593 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 23255
2026-09-26T16:42:19Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6461 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:42:19 dist-test-slave-n1hw krb5kdc[23932](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:42:19 dist-test-slave-n1hw krb5kdc[23932](info): set up 2 sockets
Sep 26 16:42:19 dist-test-slave-n1hw krb5kdc[23932](info): commencing operation
krb5kdc: starting...
W20260926 16:42:20.063967 21553 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Sep 26 16:42:20 dist-test-slave-n1hw krb5kdc[23932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:42:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:42:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
I20260926 16:42:20.184460 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:44263
--webserver_interface=127.21.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:35147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:44263
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:20.294207 23945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:20.294524 23945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:20.294622 23945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:20.298709 23945 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:42:20.298794 23945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:20.298831 23945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:20.298885 23945 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:42:20.298920 23945 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:42:20.307722 23945 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:35147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:44263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:44263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:20.308876 23945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:20.310382 23945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:20.319064 23951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:20.319088 23950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:20.319291 23945 server_base.cc:1034] running on GCE node
W20260926 16:42:20.319067 23953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:20.319921 23945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:20.320955 23945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:20.322106 23945 hybrid_clock.cc:648] HybridClock initialized: now 1790440940322071 us; error 47 us; skew 500 ppm
Sep 26 16:42:20 dist-test-slave-n1hw krb5kdc[23932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440940, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:20.324859 23945 init.cc:377] Logged in from keytab as kudu/127.21.12.126@KRBTEST.COM (short username kudu)
I20260926 16:42:20.325734 23945 webserver.cc:460] Webserver started at http://127.21.12.126:34725/ using document root <none> and password file <none>
I20260926 16:42:20.326362 23945 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:20.326422 23945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:20.326648 23945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:20.328600 23945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "890d7c824ba947b7adfa2c36c2673c72"
format_stamp: "Formatted at 2026-09-26 16:42:20 on dist-test-slave-n1hw"
server_key: "4f6736ce206d964d499ac7b98227b360"
server_key_iv: "03c40489038152d3defa3af2c6c80a07"
server_key_version: "encryptionkey@0"
I20260926 16:42:20.329159 23945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "890d7c824ba947b7adfa2c36c2673c72"
format_stamp: "Formatted at 2026-09-26 16:42:20 on dist-test-slave-n1hw"
server_key: "4f6736ce206d964d499ac7b98227b360"
server_key_iv: "03c40489038152d3defa3af2c6c80a07"
server_key_version: "encryptionkey@0"
I20260926 16:42:20.332739 23945 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260926 16:42:20.334988 23960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:20.336144 23945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 16:42:20.336304 23945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "890d7c824ba947b7adfa2c36c2673c72"
format_stamp: "Formatted at 2026-09-26 16:42:20 on dist-test-slave-n1hw"
server_key: "4f6736ce206d964d499ac7b98227b360"
server_key_iv: "03c40489038152d3defa3af2c6c80a07"
server_key_version: "encryptionkey@0"
I20260926 16:42:20.336412 23945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:20.351744 23945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:20.355000 23945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:20.355193 23945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:20.361821 23945 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.126:44263
I20260926 16:42:20.361842 24012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.126:44263 every 8 connection(s)
I20260926 16:42:20.362707 23945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:42:20.365453 24013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:20.370534 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 23945
I20260926 16:42:20.370657 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:42:20.370934 21553 external_mini_cluster.cc:1428] Setting key 654d1ce40a47bc6763b0ed93a80d994a
I20260926 16:42:20.371016 24013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 890d7c824ba947b7adfa2c36c2673c72: Bootstrap starting.
I20260926 16:42:20.373401 24013 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 890d7c824ba947b7adfa2c36c2673c72: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:20.374183 24013 log.cc:824] T 00000000000000000000000000000000 P 890d7c824ba947b7adfa2c36c2673c72: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:20.375900 24013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 890d7c824ba947b7adfa2c36c2673c72: No bootstrap required, opened a new log
Sep 26 16:42:20 dist-test-slave-n1hw krb5kdc[23932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:42:20.379401 24013 raft_consensus.cc:348] T 00000000000000000000000000000000 P 890d7c824ba947b7adfa2c36c2673c72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890d7c824ba947b7adfa2c36c2673c72" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 44263 } }
I20260926 16:42:20.379642 24013 raft_consensus.cc:374] T 00000000000000000000000000000000 P 890d7c824ba947b7adfa2c36c2673c72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:20.379721 24013 raft_consensus.cc:729] T 00000000000000000000000000000000 P 890d7c824ba947b7adfa2c36c2673c72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 890d7c824ba947b7adfa2c36c2673c72, State: Initialized, Role: FOLLOWER
I20260926 16:42:20.380185 24013 consensus_queue.cc:260] T 00000000000000000000000000000000 P 890d7c824ba947b7adfa2c36c2673c72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890d7c824ba947b7adfa2c36c2673c72" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 44263 } }
I20260926 16:42:20.380335 24013 raft_consensus.cc:388] T 00000000000000000000000000000000 P 890d7c824ba947b7adfa2c36c2673c72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:42:20.380421 24013 raft_consensus.cc:482] T 00000000000000000000000000000000 P 890d7c824ba947b7adfa2c36c2673c72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:42:20.380514 24013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 890d7c824ba947b7adfa2c36c2673c72 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:20.381332 24013 raft_consensus.cc:504] T 00000000000000000000000000000000 P 890d7c824ba947b7adfa2c36c2673c72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890d7c824ba947b7adfa2c36c2673c72" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 44263 } }
I20260926 16:42:20.381659 24013 leader_election.cc:302] T 00000000000000000000000000000000 P 890d7c824ba947b7adfa2c36c2673c72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 890d7c824ba947b7adfa2c36c2673c72; no voters:
I20260926 16:42:20.381994 24013 leader_election.cc:288] T 00000000000000000000000000000000 P 890d7c824ba947b7adfa2c36c2673c72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:42:20.382094 24018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 890d7c824ba947b7adfa2c36c2673c72 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:20.382290 24018 raft_consensus.cc:686] T 00000000000000000000000000000000 P 890d7c824ba947b7adfa2c36c2673c72 [term 1 LEADER]: Becoming Leader. State: Replica: 890d7c824ba947b7adfa2c36c2673c72, State: Running, Role: LEADER
I20260926 16:42:20.382566 24018 consensus_queue.cc:237] T 00000000000000000000000000000000 P 890d7c824ba947b7adfa2c36c2673c72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890d7c824ba947b7adfa2c36c2673c72" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 44263 } }
I20260926 16:42:20.383244 24013 sys_catalog.cc:564] T 00000000000000000000000000000000 P 890d7c824ba947b7adfa2c36c2673c72 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:42:20.383670 24018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 890d7c824ba947b7adfa2c36c2673c72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 890d7c824ba947b7adfa2c36c2673c72. Latest consensus state: current_term: 1 leader_uuid: "890d7c824ba947b7adfa2c36c2673c72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890d7c824ba947b7adfa2c36c2673c72" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 44263 } } }
I20260926 16:42:20.383805 24018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 890d7c824ba947b7adfa2c36c2673c72 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:20.384466 24019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 890d7c824ba947b7adfa2c36c2673c72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "890d7c824ba947b7adfa2c36c2673c72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890d7c824ba947b7adfa2c36c2673c72" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 44263 } } }
I20260926 16:42:20.384581 24019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 890d7c824ba947b7adfa2c36c2673c72 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:20.384883 24022 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:42:20.385191 24016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:20.372223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52468 (local address 127.21.12.126:44263)
0926 16:42:20.372648 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:20.372663 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:20.372695 (+ 32us) server_negotiation.cc:407] Connection header received
0926 16:42:20.373312 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:20.373329 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:20.373690 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:20.374063 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:20.374839 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:20.375790 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:20.376686 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:20.377149 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:20.377326 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:20.379677 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:20.379684 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:20.379701 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:20.379730 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:20.381617 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:20.382127 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:20.382133 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:20.382138 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:20.382312 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:20.382625 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:20.382628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:20.382630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:20.383056 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:20.383266 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:20.383929 (+ 663us) server_negotiation.cc:299] Negotiation successful
0926 16:42:20.384212 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":142,"threads_started":1}
I20260926 16:42:20.388926 24022 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:42:20.394186 24022 catalog_manager.cc:1347] Generated new cluster ID: f03ace03dffd4562b2a2fbb96be809b6
I20260926 16:42:20.394241 24022 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:42:20.410404 24022 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:42:20.411635 24022 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:42:20.424501 24022 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 890d7c824ba947b7adfa2c36c2673c72: Generated new TSK 0
I20260926 16:42:20.425343 24022 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab.
Entry for principal kudu/127.21.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab.
I20260926 16:42:20.455295 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.65:0
--local_ip_for_outbound_sockets=127.21.12.65
--webserver_interface=127.21.12.65
--webserver_port=0
--tserver_master_addrs=127.21.12.126:44263
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab
--principal=kudu/127.21.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:35147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:20.562865 24039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:20.563148 24039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:20.563221 24039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:20.567364 24039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:20.567451 24039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:20.567540 24039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.65
I20260926 16:42:20.572685 24039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:35147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab
--principal=kudu/127.21.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.65
--webserver_port=0
--tserver_master_addrs=127.21.12.126:44263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.65
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:20.573740 24039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:20.574771 24039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:20.584455 24047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:20.584488 24045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:20.584504 24044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:20.584597 24039 server_base.cc:1034] running on GCE node
I20260926 16:42:20.585155 24039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:20.585872 24039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:20.587077 24039 hybrid_clock.cc:648] HybridClock initialized: now 1790440940587062 us; error 35 us; skew 500 ppm
Sep 26 16:42:20 dist-test-slave-n1hw krb5kdc[23932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440940, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:20.589974 24039 init.cc:377] Logged in from keytab as kudu/127.21.12.65@KRBTEST.COM (short username kudu)
I20260926 16:42:20.590811 24039 webserver.cc:460] Webserver started at http://127.21.12.65:32993/ using document root <none> and password file <none>
I20260926 16:42:20.591357 24039 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:20.591446 24039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:20.591652 24039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:20.593590 24039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-0/data/instance:
uuid: "8645030fe8ff4a5f90391d8f38b0cda4"
format_stamp: "Formatted at 2026-09-26 16:42:20 on dist-test-slave-n1hw"
server_key: "d193d5128e47a0a2e88ca9833f301e0a"
server_key_iv: "56e8ce800b915a7ffbaeb941a926205e"
server_key_version: "encryptionkey@0"
I20260926 16:42:20.594198 24039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance:
uuid: "8645030fe8ff4a5f90391d8f38b0cda4"
format_stamp: "Formatted at 2026-09-26 16:42:20 on dist-test-slave-n1hw"
server_key: "d193d5128e47a0a2e88ca9833f301e0a"
server_key_iv: "56e8ce800b915a7ffbaeb941a926205e"
server_key_version: "encryptionkey@0"
I20260926 16:42:20.597726 24039 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:42:20.600068 24054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:20.601177 24039 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:42:20.601313 24039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-0/wal
uuid: "8645030fe8ff4a5f90391d8f38b0cda4"
format_stamp: "Formatted at 2026-09-26 16:42:20 on dist-test-slave-n1hw"
server_key: "d193d5128e47a0a2e88ca9833f301e0a"
server_key_iv: "56e8ce800b915a7ffbaeb941a926205e"
server_key_version: "encryptionkey@0"
I20260926 16:42:20.601416 24039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:20.615537 24039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:20.618583 24039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:20.618778 24039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:20.619316 24039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:42:20.620064 24039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:20.620134 24039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:20.620203 24039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:20.620256 24039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:20.628687 24039 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.65:38397
I20260926 16:42:20.628719 24167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.65:38397 every 8 connection(s)
I20260926 16:42:20.629506 24039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
I20260926 16:42:20.631541 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 24039
I20260926 16:42:20.631646 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance
I20260926 16:42:20.631913 21553 external_mini_cluster.cc:1428] Setting key fbb9ff38a46d8a88c2a683a9151a3420
Sep 26 16:42:20 dist-test-slave-n1hw krb5kdc[23932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440940, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.12.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.66@KRBTEST.COM" created.
I20260926 16:42:20.643848 24016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:20.631265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:33927 (local address 127.21.12.126:44263)
0926 16:42:20.631425 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:20.631430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:20.632677 (+ 1247us) server_negotiation.cc:407] Connection header received
0926 16:42:20.633614 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:20.633618 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:20.633674 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:20.633779 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:20.635011 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:20.635591 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:20.636823 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:20.637082 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:20.637205 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:20.640073 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:20.640076 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:20.640078 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:20.640101 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:20.641666 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:20.642336 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:20.642339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:20.642340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:20.642426 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:20.642725 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:20.642727 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:20.642728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:20.643001 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:20.643088 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:20.643573 (+ 485us) server_negotiation.cc:299] Negotiation successful
0926 16:42:20.643713 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 16:42:20.644659 24170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:20.631764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:44263 (local address 127.21.12.65:33927)
0926 16:42:20.632520 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:20.632573 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:20.633392 (+ 819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:20.633960 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:20.633967 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:20.634370 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:20.634852 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:20.634866 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:20.635727 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:20.635731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:20.636676 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:20.636685 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:20.637242 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:20.637247 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:20.637433 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:20.638075 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:20.638092 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:20.639881 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:20.641944 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:20.641954 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:20.641964 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:20.642200 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:20.642510 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:20.642512 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:20.642514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:20.642637 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:20.643098 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:20.643102 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:20.643349 (+ 247us) client_negotiation.cc:769] Sending connection context
0926 16:42:20.643522 (+ 173us) client_negotiation.cc:240] Negotiation successful
0926 16:42:20.643868 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":141,"threads_started":1}
I20260926 16:42:20.646001 24168 heartbeater.cc:347] Connected to a master server at 127.21.12.126:44263
I20260926 16:42:20.646231 24168 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:20.646733 24168 heartbeater.cc:511] Master 127.21.12.126:44263 requested a full tablet report, sending...
I20260926 16:42:20.648356 23977 ts_manager.cc:194] Registered new tserver with Master: 8645030fe8ff4a5f90391d8f38b0cda4 (127.21.12.65:38397)
I20260926 16:42:20.650039 23977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.65@KRBTEST.COM'} at 127.21.12.65:33927
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab.
Entry for principal kudu/127.21.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab.
I20260926 16:42:20.660969 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.66:0
--local_ip_for_outbound_sockets=127.21.12.66
--webserver_interface=127.21.12.66
--webserver_port=0
--tserver_master_addrs=127.21.12.126:44263
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab
--principal=kudu/127.21.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:35147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:20.769253 24173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:20.769483 24173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:20.769523 24173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:20.773589 24173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:20.773640 24173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:20.773739 24173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.66
I20260926 16:42:20.778772 24173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:35147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab
--principal=kudu/127.21.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.66
--webserver_port=0
--tserver_master_addrs=127.21.12.126:44263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.66
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:20.779732 24173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:20.780717 24173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:20.790025 24179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:20.790030 24178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:20.790030 24181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:20.790208 24173 server_base.cc:1034] running on GCE node
I20260926 16:42:20.790578 24173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:20.791267 24173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:20.792464 24173 hybrid_clock.cc:648] HybridClock initialized: now 1790440940792438 us; error 42 us; skew 500 ppm
Sep 26 16:42:20 dist-test-slave-n1hw krb5kdc[23932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440940, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:20.795390 24173 init.cc:377] Logged in from keytab as kudu/127.21.12.66@KRBTEST.COM (short username kudu)
I20260926 16:42:20.796317 24173 webserver.cc:460] Webserver started at http://127.21.12.66:41719/ using document root <none> and password file <none>
I20260926 16:42:20.796916 24173 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:20.797012 24173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:20.797247 24173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:20.799412 24173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-1/data/instance:
uuid: "a3921909c6194fef91a8f711743dab01"
format_stamp: "Formatted at 2026-09-26 16:42:20 on dist-test-slave-n1hw"
server_key: "eba7a475b398805e75951c9d2b0f2711"
server_key_iv: "d90b825430021ca31f39167487f12b9f"
server_key_version: "encryptionkey@0"
I20260926 16:42:20.800001 24173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance:
uuid: "a3921909c6194fef91a8f711743dab01"
format_stamp: "Formatted at 2026-09-26 16:42:20 on dist-test-slave-n1hw"
server_key: "eba7a475b398805e75951c9d2b0f2711"
server_key_iv: "d90b825430021ca31f39167487f12b9f"
server_key_version: "encryptionkey@0"
I20260926 16:42:20.803802 24173 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 16:42:20.806408 24188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:20.807588 24173 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:42:20.807749 24173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-1/wal
uuid: "a3921909c6194fef91a8f711743dab01"
format_stamp: "Formatted at 2026-09-26 16:42:20 on dist-test-slave-n1hw"
server_key: "eba7a475b398805e75951c9d2b0f2711"
server_key_iv: "d90b825430021ca31f39167487f12b9f"
server_key_version: "encryptionkey@0"
I20260926 16:42:20.807858 24173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:20.823837 24173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:20.827306 24173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:20.827519 24173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:20.828164 24173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:42:20.829071 24173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:20.829156 24173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:20.829252 24173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:20.829308 24173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:20.839196 24173 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.66:34387
I20260926 16:42:20.839217 24301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.66:34387 every 8 connection(s)
I20260926 16:42:20.840126 24173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
I20260926 16:42:20.846601 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 24173
I20260926 16:42:20.846725 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance
I20260926 16:42:20.847012 21553 external_mini_cluster.cc:1428] Setting key c18d8e5f99b2aa745fbf36b701250d3b
Sep 26 16:42:20 dist-test-slave-n1hw krb5kdc[23932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440940, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.66@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:42:20.855086 24016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:20.842160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:44009 (local address 127.21.12.126:44263)
0926 16:42:20.842329 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:20.842333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:20.843446 (+ 1113us) server_negotiation.cc:407] Connection header received
0926 16:42:20.844230 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:20.844233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:20.844279 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:20.844356 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:20.845490 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:20.846262 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:20.847779 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:20.848020 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:20.848108 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:20.850868 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:20.850872 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:20.850877 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:20.850914 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:20.852710 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:20.853292 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:20.853298 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:20.853302 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:20.853399 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:20.853911 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:20.853914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:20.853915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:20.854088 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:20.854168 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:20.854882 (+ 714us) server_negotiation.cc:299] Negotiation successful
0926 16:42:20.854975 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 16:42:20.856427 24304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:20.842559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:44263 (local address 127.21.12.66:44009)
0926 16:42:20.843230 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:20.843271 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:20.844024 (+ 753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:20.844521 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:20.844527 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:20.844859 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:20.845326 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:20.845338 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:20.846399 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:20.846402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:20.847653 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:20.847664 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:20.848189 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:20.848194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:20.848443 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:20.849149 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:20.849166 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:20.850691 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:20.852880 (+ 2189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:20.852887 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:20.852903 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:20.853179 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:20.853503 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:20.853505 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:20.853507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:20.853786 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:20.854205 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:20.854212 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:20.854564 (+ 352us) client_negotiation.cc:769] Sending connection context
0926 16:42:20.854843 (+ 279us) client_negotiation.cc:240] Negotiation successful
0926 16:42:20.855238 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":109,"threads_started":1}
I20260926 16:42:20.857661 24302 heartbeater.cc:347] Connected to a master server at 127.21.12.126:44263
I20260926 16:42:20.857956 24302 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.21.12.67@KRBTEST.COM; defaulting to no policy
I20260926 16:42:20.858446 24302 heartbeater.cc:511] Master 127.21.12.126:44263 requested a full tablet report, sending...
I20260926 16:42:20.859503 23977 ts_manager.cc:194] Registered new tserver with Master: a3921909c6194fef91a8f711743dab01 (127.21.12.66:34387)
I20260926 16:42:20.860037 23977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.66@KRBTEST.COM'} at 127.21.12.66:44009
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab.
Entry for principal kudu/127.21.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab.
I20260926 16:42:20.874943 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.67:0
--local_ip_for_outbound_sockets=127.21.12.67
--webserver_interface=127.21.12.67
--webserver_port=0
--tserver_master_addrs=127.21.12.126:44263
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab
--principal=kudu/127.21.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:35147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:20.984052 24307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:20.984279 24307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:20.984320 24307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:20.988286 24307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:20.988334 24307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:20.988389 24307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.67
I20260926 16:42:20.993371 24307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:35147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab
--principal=kudu/127.21.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.67
--webserver_port=0
--tserver_master_addrs=127.21.12.126:44263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.67
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:20.994277 24307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:20.995136 24307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:21.004048 24312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:21.004094 24313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:21.004132 24315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:21.004376 24307 server_base.cc:1034] running on GCE node
I20260926 16:42:21.004732 24307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:21.005304 24307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:21.006459 24307 hybrid_clock.cc:648] HybridClock initialized: now 1790440941006428 us; error 42 us; skew 500 ppm
Sep 26 16:42:21 dist-test-slave-n1hw krb5kdc[23932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440941, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:21.009137 24307 init.cc:377] Logged in from keytab as kudu/127.21.12.67@KRBTEST.COM (short username kudu)
I20260926 16:42:21.010008 24307 webserver.cc:460] Webserver started at http://127.21.12.67:37397/ using document root <none> and password file <none>
I20260926 16:42:21.010491 24307 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:21.010547 24307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:21.010694 24307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:21.012612 24307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-2/data/instance:
uuid: "ac1406f836fe4d798b8aec6510478648"
format_stamp: "Formatted at 2026-09-26 16:42:21 on dist-test-slave-n1hw"
server_key: "21ac4e55edb333a52e6292cd5820fa2f"
server_key_iv: "f84d8a2638bcf60b2c0eefe7fed1d293"
server_key_version: "encryptionkey@0"
I20260926 16:42:21.013087 24307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance:
uuid: "ac1406f836fe4d798b8aec6510478648"
format_stamp: "Formatted at 2026-09-26 16:42:21 on dist-test-slave-n1hw"
server_key: "21ac4e55edb333a52e6292cd5820fa2f"
server_key_iv: "f84d8a2638bcf60b2c0eefe7fed1d293"
server_key_version: "encryptionkey@0"
I20260926 16:42:21.016505 24307 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 16:42:21.018711 24322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:21.019894 24307 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:42:21.020008 24307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-2/wal
uuid: "ac1406f836fe4d798b8aec6510478648"
format_stamp: "Formatted at 2026-09-26 16:42:21 on dist-test-slave-n1hw"
server_key: "21ac4e55edb333a52e6292cd5820fa2f"
server_key_iv: "f84d8a2638bcf60b2c0eefe7fed1d293"
server_key_version: "encryptionkey@0"
I20260926 16:42:21.020102 24307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:21.043785 24307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:21.047137 24307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:21.047353 24307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:21.047951 24307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:42:21.048779 24307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:21.048823 24307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:21.048902 24307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:21.048934 24307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:21.058290 24307 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.67:45195
I20260926 16:42:21.058308 24435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.67:45195 every 8 connection(s)
I20260926 16:42:21.059180 24307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
I20260926 16:42:21.060642 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 24307
I20260926 16:42:21.060794 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance
I20260926 16:42:21.061111 21553 external_mini_cluster.cc:1428] Setting key 0b86647fc799198f0448b8e7720ad005
Sep 26 16:42:21 dist-test-slave-n1hw krb5kdc[23932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440941, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.67@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:42:21.072947 24016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:21.061066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:56059 (local address 127.21.12.126:44263)
0926 16:42:21.061206 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:21.061210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:21.062821 (+ 1611us) server_negotiation.cc:407] Connection header received
0926 16:42:21.063823 (+ 1002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:21.063826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:21.063869 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:21.063934 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:21.065098 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:21.065609 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:21.066864 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:21.067099 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:21.067186 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:21.069571 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:21.069574 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:21.069576 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:21.069600 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:21.071141 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:21.071593 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:21.071595 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:21.071596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:21.071671 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:21.071997 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:21.072000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:21.072001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:21.072171 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:21.072253 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:21.072709 (+ 456us) server_negotiation.cc:299] Negotiation successful
0926 16:42:21.072831 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 16:42:21.073956 24438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:21.061572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:44263 (local address 127.21.12.67:56059)
0926 16:42:21.062637 (+ 1065us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:21.062706 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:21.063622 (+ 916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:21.064107 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:21.064113 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:21.064463 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:21.064970 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:21.064983 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:21.065707 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:21.065709 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:21.066710 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:21.066720 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:21.067227 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:21.067230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:21.067382 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:21.067987 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:21.068003 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:21.069427 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:21.071245 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:21.071250 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:21.071260 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:21.071503 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:21.071770 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:21.071772 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:21.071774 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:21.071905 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:21.072261 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:21.072266 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:21.072496 (+ 230us) client_negotiation.cc:769] Sending connection context
0926 16:42:21.072649 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 16:42:21.072887 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":865,"thread_start_us":741,"threads_started":1}
I20260926 16:42:21.075073 24436 heartbeater.cc:347] Connected to a master server at 127.21.12.126:44263
I20260926 16:42:21.075331 24436 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:21.075789 24436 heartbeater.cc:511] Master 127.21.12.126:44263 requested a full tablet report, sending...
I20260926 16:42:21.076761 23977 ts_manager.cc:194] Registered new tserver with Master: ac1406f836fe4d798b8aec6510478648 (127.21.12.67:45195)
I20260926 16:42:21.077312 23977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.67@KRBTEST.COM'} at 127.21.12.67:56059
I20260926 16:42:21.086833 21553 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 16:42:21.099068 24016 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:21.097682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52472 (local address 127.21.12.126:44263)
0926 16:42:21.097830 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:21.097833 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:21.098029 (+ 196us) server_negotiation.cc:407] Connection header received
0926 16:42:21.098211 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:21.098214 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:21.098257 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:21.098327 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:21.098975 (+ 648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52472: BlockingRecv error: recv got EOF from 127.0.0.1:52472 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 16:42:21.100095 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 24039
I20260926 16:42:21.105808 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 24173
I20260926 16:42:21.111071 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 24307
I20260926 16:42:21.116221 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 23945
2026-09-26T16:42:21Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2105 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440924937351-21553-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:42:21 dist-test-slave-n1hw krb5kdc[24448](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:42:21 dist-test-slave-n1hw krb5kdc[24448](info): set up 2 sockets
Sep 26 16:42:21 dist-test-slave-n1hw krb5kdc[24448](info): commencing operation
krb5kdc: starting...
W20260926 16:42:22.147740 21553 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Sep 26 16:42:22 dist-test-slave-n1hw krb5kdc[24448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:42:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:42:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
I20260926 16:42:22.265126 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:38811
--webserver_interface=127.21.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:35903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:38811
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:22.373570 24461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:22.373904 24461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:22.373986 24461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:22.378399 24461 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:42:22.378481 24461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:22.378518 24461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:22.378575 24461 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:42:22.378626 24461 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:42:22.383931 24461 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:35903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:38811
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:38811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:22.384920 24461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:22.385953 24461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:22.394094 24467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:22.394194 24461 server_base.cc:1034] running on GCE node
W20260926 16:42:22.394140 24466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:22.394085 24469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:22.394928 24461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:22.395941 24461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:22.397128 24461 hybrid_clock.cc:648] HybridClock initialized: now 1790440942397112 us; error 37 us; skew 500 ppm
Sep 26 16:42:22 dist-test-slave-n1hw krb5kdc[24448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440942, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:22.422928 24461 init.cc:377] Logged in from keytab as kudu/127.21.12.126@KRBTEST.COM (short username kudu)
I20260926 16:42:22.423942 24461 webserver.cc:460] Webserver started at http://127.21.12.126:45377/ using document root <none> and password file <none>
I20260926 16:42:22.424475 24461 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:22.424536 24461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:22.424700 24461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:22.426764 24461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "3f3350cd5044432e964440600ac183c2"
format_stamp: "Formatted at 2026-09-26 16:42:22 on dist-test-slave-n1hw"
server_key: "5f3616677c6edee6ec58ea5e2c3f1638"
server_key_iv: "18e6e5d85901f4623ae68cd934f50117"
server_key_version: "encryptionkey@0"
I20260926 16:42:22.427284 24461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "3f3350cd5044432e964440600ac183c2"
format_stamp: "Formatted at 2026-09-26 16:42:22 on dist-test-slave-n1hw"
server_key: "5f3616677c6edee6ec58ea5e2c3f1638"
server_key_iv: "18e6e5d85901f4623ae68cd934f50117"
server_key_version: "encryptionkey@0"
I20260926 16:42:22.430931 24461 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260926 16:42:22.433349 24476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:22.434685 24461 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:42:22.434798 24461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "3f3350cd5044432e964440600ac183c2"
format_stamp: "Formatted at 2026-09-26 16:42:22 on dist-test-slave-n1hw"
server_key: "5f3616677c6edee6ec58ea5e2c3f1638"
server_key_iv: "18e6e5d85901f4623ae68cd934f50117"
server_key_version: "encryptionkey@0"
I20260926 16:42:22.434882 24461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:22.452992 24461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:22.456081 24461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:22.456225 24461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:22.463030 24461 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.126:38811
I20260926 16:42:22.463301 24528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.126:38811 every 8 connection(s)
I20260926 16:42:22.463958 24461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:42:22.466609 24529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:22.470801 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 24461
I20260926 16:42:22.470921 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:42:22.471196 21553 external_mini_cluster.cc:1428] Setting key 751c3c4d5644f4ccc672c07406153c12
I20260926 16:42:22.472455 24529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f3350cd5044432e964440600ac183c2: Bootstrap starting.
I20260926 16:42:22.474833 24529 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f3350cd5044432e964440600ac183c2: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:22.475549 24529 log.cc:824] T 00000000000000000000000000000000 P 3f3350cd5044432e964440600ac183c2: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:22.477587 24529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f3350cd5044432e964440600ac183c2: No bootstrap required, opened a new log
Sep 26 16:42:22 dist-test-slave-n1hw krb5kdc[24448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:42:22.482125 24529 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f3350cd5044432e964440600ac183c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3350cd5044432e964440600ac183c2" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 38811 } }
I20260926 16:42:22.482373 24529 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f3350cd5044432e964440600ac183c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:22.482435 24529 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f3350cd5044432e964440600ac183c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f3350cd5044432e964440600ac183c2, State: Initialized, Role: FOLLOWER
I20260926 16:42:22.482903 24529 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f3350cd5044432e964440600ac183c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3350cd5044432e964440600ac183c2" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 38811 } }
I20260926 16:42:22.483026 24529 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f3350cd5044432e964440600ac183c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:42:22.483103 24529 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f3350cd5044432e964440600ac183c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:42:22.483229 24529 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f3350cd5044432e964440600ac183c2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:22.484023 24529 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f3350cd5044432e964440600ac183c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3350cd5044432e964440600ac183c2" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 38811 } }
I20260926 16:42:22.484396 24529 leader_election.cc:302] T 00000000000000000000000000000000 P 3f3350cd5044432e964440600ac183c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f3350cd5044432e964440600ac183c2; no voters:
I20260926 16:42:22.484725 24529 leader_election.cc:288] T 00000000000000000000000000000000 P 3f3350cd5044432e964440600ac183c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:42:22.484872 24534 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f3350cd5044432e964440600ac183c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:22.485095 24534 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f3350cd5044432e964440600ac183c2 [term 1 LEADER]: Becoming Leader. State: Replica: 3f3350cd5044432e964440600ac183c2, State: Running, Role: LEADER
I20260926 16:42:22.485471 24534 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f3350cd5044432e964440600ac183c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3350cd5044432e964440600ac183c2" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 38811 } }
I20260926 16:42:22.485934 24529 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f3350cd5044432e964440600ac183c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:42:22.486233 24532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:22.472412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56544 (local address 127.21.12.126:38811)
0926 16:42:22.472832 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:22.472847 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:22.472880 (+ 33us) server_negotiation.cc:407] Connection header received
0926 16:42:22.473479 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:22.473494 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:22.473870 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:22.474140 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:22.475039 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:22.476151 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:22.477303 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:22.477839 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:22.477959 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:22.480591 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:22.480599 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:22.480618 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:22.480655 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:22.482672 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:22.483182 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:22.483188 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:22.483195 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:22.483418 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:22.483728 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:22.483734 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:22.483736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:22.484127 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:22.484327 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:22.484668 (+ 341us) server_negotiation.cc:299] Negotiation successful
0926 16:42:22.485168 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":161,"threads_started":1}
I20260926 16:42:22.486796 24536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f3350cd5044432e964440600ac183c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f3350cd5044432e964440600ac183c2. Latest consensus state: current_term: 1 leader_uuid: "3f3350cd5044432e964440600ac183c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3350cd5044432e964440600ac183c2" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 38811 } } }
I20260926 16:42:22.486920 24536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f3350cd5044432e964440600ac183c2 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:22.487165 24535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f3350cd5044432e964440600ac183c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f3350cd5044432e964440600ac183c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3350cd5044432e964440600ac183c2" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 38811 } } }
I20260926 16:42:22.487285 24535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f3350cd5044432e964440600ac183c2 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:22.487505 24543 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:42:22.489502 21553 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:42:22.490957 24543 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:42:22.496984 24543 catalog_manager.cc:1347] Generated new cluster ID: 0a7fe3befbcb40de8aca9f9e7dad949b
I20260926 16:42:22.497056 24543 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:42:22.513345 24532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:22.507358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56550 (local address 127.21.12.126:38811)
0926 16:42:22.507470 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:22.507474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:22.507703 (+ 229us) server_negotiation.cc:407] Connection header received
0926 16:42:22.507896 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:22.507899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:22.507950 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:22.508003 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:22.508669 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:22.509521 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:22.510771 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:22.511129 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:22.511202 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:22.512282 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:22.512286 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:22.512289 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:22.512841 (+ 552us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:22.512874 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:22.512893 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:22.512954 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:22.513129 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 16:42:22.513187 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 16:42:22.522930 24543 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:42:22.523859 24543 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:42:22.529103 24543 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f3350cd5044432e964440600ac183c2: Generated new TSK 0
I20260926 16:42:22.529820 24543 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:42:22.538821 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 24461
2026-09-26T16:42:22Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1419 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:42:22 dist-test-slave-n1hw krb5kdc[24562](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:42:22 dist-test-slave-n1hw krb5kdc[24562](info): set up 2 sockets
Sep 26 16:42:22 dist-test-slave-n1hw krb5kdc[24562](info): commencing operation
krb5kdc: starting...
W20260926 16:42:23.587913 21553 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Sep 26 16:42:23 dist-test-slave-n1hw krb5kdc[24562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:42:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:42:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
I20260926 16:42:23.704380 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:37663
--webserver_interface=127.21.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:32825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:37663
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:23.811219 24575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:23.811439 24575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:23.811476 24575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:23.815650 24575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:42:23.815699 24575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:23.815719 24575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:23.815735 24575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:42:23.815757 24575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:42:23.820756 24575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:32825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:37663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:37663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:23.821630 24575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:23.822566 24575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:23.830783 24583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:23.830828 24575 server_base.cc:1034] running on GCE node
W20260926 16:42:23.830737 24581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:23.830744 24580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:23.831427 24575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:23.832464 24575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:23.833654 24575 hybrid_clock.cc:648] HybridClock initialized: now 1790440943833623 us; error 43 us; skew 500 ppm
Sep 26 16:42:23 dist-test-slave-n1hw krb5kdc[24562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440943, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:23.836406 24575 init.cc:377] Logged in from keytab as kudu/127.21.12.126@KRBTEST.COM (short username kudu)
I20260926 16:42:23.837261 24575 webserver.cc:460] Webserver started at http://127.21.12.126:43697/ using document root <none> and password file <none>
I20260926 16:42:23.837837 24575 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:23.837922 24575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:23.838138 24575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:23.840065 24575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "19d9e22aa297430b9a4d00c1dd3c9cdc"
format_stamp: "Formatted at 2026-09-26 16:42:23 on dist-test-slave-n1hw"
server_key: "22a5622adc0baee661a2e7cd857206d4"
server_key_iv: "ea5eb56d9728f0dfc5d0a69cac0643c0"
server_key_version: "encryptionkey@0"
I20260926 16:42:23.840608 24575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "19d9e22aa297430b9a4d00c1dd3c9cdc"
format_stamp: "Formatted at 2026-09-26 16:42:23 on dist-test-slave-n1hw"
server_key: "22a5622adc0baee661a2e7cd857206d4"
server_key_iv: "ea5eb56d9728f0dfc5d0a69cac0643c0"
server_key_version: "encryptionkey@0"
I20260926 16:42:23.844202 24575 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260926 16:42:23.846630 24590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:23.847889 24575 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260926 16:42:23.848047 24575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "19d9e22aa297430b9a4d00c1dd3c9cdc"
format_stamp: "Formatted at 2026-09-26 16:42:23 on dist-test-slave-n1hw"
server_key: "22a5622adc0baee661a2e7cd857206d4"
server_key_iv: "ea5eb56d9728f0dfc5d0a69cac0643c0"
server_key_version: "encryptionkey@0"
I20260926 16:42:23.848156 24575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:23.865571 24575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:23.868875 24575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:23.869087 24575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:23.876266 24575 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.126:37663
I20260926 16:42:23.876566 24642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.126:37663 every 8 connection(s)
I20260926 16:42:23.877188 24575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:42:23.879561 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 24575
I20260926 16:42:23.879706 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:42:23.879956 24643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:23.879974 21553 external_mini_cluster.cc:1428] Setting key 088f4800f62184cc4b88cde7af582cfe
I20260926 16:42:23.886929 24643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19d9e22aa297430b9a4d00c1dd3c9cdc: Bootstrap starting.
I20260926 16:42:23.889137 24643 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19d9e22aa297430b9a4d00c1dd3c9cdc: Neither blocks nor log segments found. Creating new log.
Sep 26 16:42:23 dist-test-slave-n1hw krb5kdc[24562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:42:23.889955 24643 log.cc:824] T 00000000000000000000000000000000 P 19d9e22aa297430b9a4d00c1dd3c9cdc: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:23.892020 24643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19d9e22aa297430b9a4d00c1dd3c9cdc: No bootstrap required, opened a new log
I20260926 16:42:23.894582 24646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:23.881339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59182 (local address 127.21.12.126:37663)
0926 16:42:23.881720 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:23.881737 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:23.881810 (+ 73us) server_negotiation.cc:407] Connection header received
0926 16:42:23.882619 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:23.882642 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:23.883042 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:23.883412 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:23.884101 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:23.885586 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:23.886567 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:23.887229 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:23.887384 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:23.889949 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:23.889956 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:23.889973 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:23.890005 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:23.891747 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:23.892205 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:23.892210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:23.892214 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:23.892354 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:23.892636 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:23.892638 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:23.892639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:23.892979 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:23.893118 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:23.893487 (+ 369us) server_negotiation.cc:299] Negotiation successful
0926 16:42:23.893856 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":96,"threads_started":1}
I20260926 16:42:23.895983 24643 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19d9e22aa297430b9a4d00c1dd3c9cdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d9e22aa297430b9a4d00c1dd3c9cdc" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 37663 } }
I20260926 16:42:23.896250 24643 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19d9e22aa297430b9a4d00c1dd3c9cdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:23.896307 24643 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19d9e22aa297430b9a4d00c1dd3c9cdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19d9e22aa297430b9a4d00c1dd3c9cdc, State: Initialized, Role: FOLLOWER
I20260926 16:42:23.896713 24643 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19d9e22aa297430b9a4d00c1dd3c9cdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d9e22aa297430b9a4d00c1dd3c9cdc" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 37663 } }
I20260926 16:42:23.896874 24643 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19d9e22aa297430b9a4d00c1dd3c9cdc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:42:23.896958 24643 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19d9e22aa297430b9a4d00c1dd3c9cdc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:42:23.897078 24643 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19d9e22aa297430b9a4d00c1dd3c9cdc [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:23.897953 24643 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19d9e22aa297430b9a4d00c1dd3c9cdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d9e22aa297430b9a4d00c1dd3c9cdc" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 37663 } }
I20260926 16:42:23.898263 24643 leader_election.cc:302] T 00000000000000000000000000000000 P 19d9e22aa297430b9a4d00c1dd3c9cdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19d9e22aa297430b9a4d00c1dd3c9cdc; no voters:
I20260926 16:42:23.898559 24643 leader_election.cc:288] T 00000000000000000000000000000000 P 19d9e22aa297430b9a4d00c1dd3c9cdc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:42:23.898706 24648 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19d9e22aa297430b9a4d00c1dd3c9cdc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:23.898924 24648 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19d9e22aa297430b9a4d00c1dd3c9cdc [term 1 LEADER]: Becoming Leader. State: Replica: 19d9e22aa297430b9a4d00c1dd3c9cdc, State: Running, Role: LEADER
I20260926 16:42:23.899207 24648 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19d9e22aa297430b9a4d00c1dd3c9cdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d9e22aa297430b9a4d00c1dd3c9cdc" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 37663 } }
I20260926 16:42:23.899714 24643 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19d9e22aa297430b9a4d00c1dd3c9cdc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:42:23.901029 24649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19d9e22aa297430b9a4d00c1dd3c9cdc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19d9e22aa297430b9a4d00c1dd3c9cdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d9e22aa297430b9a4d00c1dd3c9cdc" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 37663 } } }
I20260926 16:42:23.901093 24650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19d9e22aa297430b9a4d00c1dd3c9cdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19d9e22aa297430b9a4d00c1dd3c9cdc. Latest consensus state: current_term: 1 leader_uuid: "19d9e22aa297430b9a4d00c1dd3c9cdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d9e22aa297430b9a4d00c1dd3c9cdc" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 37663 } } }
I20260926 16:42:23.901157 24649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19d9e22aa297430b9a4d00c1dd3c9cdc [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:23.901237 24650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19d9e22aa297430b9a4d00c1dd3c9cdc [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:23.901572 24657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:42:23.904526 24657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:42:23.909623 24657 catalog_manager.cc:1347] Generated new cluster ID: 0c59b36315b749b59aa145d3959379d7
I20260926 16:42:23.909686 24657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:42:23.919723 24657 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:42:23.920841 24657 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:42:23.927706 24657 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19d9e22aa297430b9a4d00c1dd3c9cdc: Generated new TSK 0
I20260926 16:42:23.928428 24657 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab.
Entry for principal kudu/127.21.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab.
I20260926 16:42:23.962984 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.65:0
--local_ip_for_outbound_sockets=127.21.12.65
--webserver_interface=127.21.12.65
--webserver_port=0
--tserver_master_addrs=127.21.12.126:37663
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab
--principal=kudu/127.21.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:32825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:24.076237 24669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:24.076468 24669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:24.076508 24669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:24.080565 24669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:24.080614 24669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:24.080708 24669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.65
I20260926 16:42:24.085865 24669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:32825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab
--principal=kudu/127.21.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.65
--webserver_port=0
--tserver_master_addrs=127.21.12.126:37663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.65
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:24.086802 24669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:24.087767 24669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:24.096781 24674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:24.096800 24675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:24.096798 24677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:24.097012 24669 server_base.cc:1034] running on GCE node
I20260926 16:42:24.097420 24669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:24.098091 24669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:24.099272 24669 hybrid_clock.cc:648] HybridClock initialized: now 1790440944099250 us; error 63 us; skew 500 ppm
Sep 26 16:42:24 dist-test-slave-n1hw krb5kdc[24562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440944, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:24.102152 24669 init.cc:377] Logged in from keytab as kudu/127.21.12.65@KRBTEST.COM (short username kudu)
I20260926 16:42:24.102986 24669 webserver.cc:460] Webserver started at http://127.21.12.65:35079/ using document root <none> and password file <none>
I20260926 16:42:24.103521 24669 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:24.103608 24669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:24.103829 24669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:24.105830 24669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-0/data/instance:
uuid: "8227198f76074109b1e49abca9a644d6"
format_stamp: "Formatted at 2026-09-26 16:42:24 on dist-test-slave-n1hw"
server_key: "401e9ec0bac67ef40f8f9b567c830da9"
server_key_iv: "5a2aa4a71163b2c78959ae98ca1f4315"
server_key_version: "encryptionkey@0"
I20260926 16:42:24.106410 24669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance:
uuid: "8227198f76074109b1e49abca9a644d6"
format_stamp: "Formatted at 2026-09-26 16:42:24 on dist-test-slave-n1hw"
server_key: "401e9ec0bac67ef40f8f9b567c830da9"
server_key_iv: "5a2aa4a71163b2c78959ae98ca1f4315"
server_key_version: "encryptionkey@0"
I20260926 16:42:24.110023 24669 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:42:24.112313 24684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:24.113466 24669 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:42:24.113598 24669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-0/wal
uuid: "8227198f76074109b1e49abca9a644d6"
format_stamp: "Formatted at 2026-09-26 16:42:24 on dist-test-slave-n1hw"
server_key: "401e9ec0bac67ef40f8f9b567c830da9"
server_key_iv: "5a2aa4a71163b2c78959ae98ca1f4315"
server_key_version: "encryptionkey@0"
I20260926 16:42:24.113705 24669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:24.135309 24669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:24.138085 24669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:24.138279 24669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:24.138825 24669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:42:24.139606 24669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:24.139685 24669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:24.139758 24669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:24.139792 24669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:24.148516 24669 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.65:46087
I20260926 16:42:24.148543 24797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.65:46087 every 8 connection(s)
I20260926 16:42:24.149364 24669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
Sep 26 16:42:24 dist-test-slave-n1hw krb5kdc[24562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440944, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:42:24.159421 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 24669
I20260926 16:42:24.159538 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance
I20260926 16:42:24.159798 21553 external_mini_cluster.cc:1428] Setting key 6a34b4ea90ec54de25a5b17c56a92783
I20260926 16:42:24.162767 24646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:24.151117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:33905 (local address 127.21.12.126:37663)
0926 16:42:24.151263 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:24.151273 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:24.152167 (+ 894us) server_negotiation.cc:407] Connection header received
0926 16:42:24.153544 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:24.153547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:24.153592 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:24.153663 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:24.154754 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:24.155277 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:24.156328 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:24.156552 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:24.156626 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:24.159018 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:24.159026 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:24.159029 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:24.159053 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:24.160689 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:24.161247 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:24.161253 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:24.161254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:24.161336 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:24.161677 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:24.161680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:24.161682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:24.161919 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:24.162023 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:24.162487 (+ 464us) server_negotiation.cc:299] Negotiation successful
0926 16:42:24.162604 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 16:42:24.163476 24800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:24.151518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:37663 (local address 127.21.12.65:33905)
0926 16:42:24.151967 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:24.152006 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:24.153346 (+ 1340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:24.153833 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:24.153839 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:24.154192 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:24.154628 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:24.154641 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:24.155389 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:24.155391 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:24.156228 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:24.156236 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:24.156669 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:24.156671 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:24.156800 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:24.157405 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:24.157420 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:24.158855 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:24.160805 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:24.160810 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:24.160820 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:24.161148 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:24.161441 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:24.161444 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:24.161446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:24.161574 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:24.162030 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:24.162038 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:24.162255 (+ 217us) client_negotiation.cc:769] Sending connection context
0926 16:42:24.162434 (+ 179us) client_negotiation.cc:240] Negotiation successful
0926 16:42:24.162726 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":109,"threads_started":1}
I20260926 16:42:24.164629 24798 heartbeater.cc:347] Connected to a master server at 127.21.12.126:37663
I20260926 16:42:24.164880 24798 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:24.165319 24798 heartbeater.cc:511] Master 127.21.12.126:37663 requested a full tablet report, sending...
I20260926 16:42:24.166742 24607 ts_manager.cc:194] Registered new tserver with Master: 8227198f76074109b1e49abca9a644d6 (127.21.12.65:46087)
I20260926 16:42:24.167815 24607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.65@KRBTEST.COM'} at 127.21.12.65:33905
WARNING: no policy specified for kudu/127.21.12.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab.
Entry for principal kudu/127.21.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab.
I20260926 16:42:24.186460 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.66:0
--local_ip_for_outbound_sockets=127.21.12.66
--webserver_interface=127.21.12.66
--webserver_port=0
--tserver_master_addrs=127.21.12.126:37663
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab
--principal=kudu/127.21.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:32825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:24.291463 24803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:24.291679 24803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:24.291754 24803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:24.295923 24803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:24.295969 24803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:24.296022 24803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.66
I20260926 16:42:24.301831 24803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:32825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab
--principal=kudu/127.21.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.66
--webserver_port=0
--tserver_master_addrs=127.21.12.126:37663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.66
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:24.303000 24803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:24.304280 24803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:24.314822 24809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:24.314812 24811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:24.314815 24808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:24.315109 24803 server_base.cc:1034] running on GCE node
I20260926 16:42:24.315455 24803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:24.316066 24803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:24.317261 24803 hybrid_clock.cc:648] HybridClock initialized: now 1790440944317227 us; error 49 us; skew 500 ppm
Sep 26 16:42:24 dist-test-slave-n1hw krb5kdc[24562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440944, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:24.319804 24803 init.cc:377] Logged in from keytab as kudu/127.21.12.66@KRBTEST.COM (short username kudu)
I20260926 16:42:24.320605 24803 webserver.cc:460] Webserver started at http://127.21.12.66:46711/ using document root <none> and password file <none>
I20260926 16:42:24.321175 24803 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:24.321264 24803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:24.321468 24803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:24.323374 24803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-1/data/instance:
uuid: "920d365e25054fdcbde52bb31f57d6d1"
format_stamp: "Formatted at 2026-09-26 16:42:24 on dist-test-slave-n1hw"
server_key: "8fa8e916f4c3be5b4524d462f884403d"
server_key_iv: "277e2da4d4dfee592174a46822fb58b2"
server_key_version: "encryptionkey@0"
I20260926 16:42:24.323906 24803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance:
uuid: "920d365e25054fdcbde52bb31f57d6d1"
format_stamp: "Formatted at 2026-09-26 16:42:24 on dist-test-slave-n1hw"
server_key: "8fa8e916f4c3be5b4524d462f884403d"
server_key_iv: "277e2da4d4dfee592174a46822fb58b2"
server_key_version: "encryptionkey@0"
I20260926 16:42:24.327348 24803 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 16:42:24.329530 24818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:24.330667 24803 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:42:24.330809 24803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-1/wal
uuid: "920d365e25054fdcbde52bb31f57d6d1"
format_stamp: "Formatted at 2026-09-26 16:42:24 on dist-test-slave-n1hw"
server_key: "8fa8e916f4c3be5b4524d462f884403d"
server_key_iv: "277e2da4d4dfee592174a46822fb58b2"
server_key_version: "encryptionkey@0"
I20260926 16:42:24.330912 24803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:24.341307 24803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:24.343915 24803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:24.344103 24803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:24.344615 24803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:42:24.345355 24803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:24.345423 24803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:24.345491 24803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:24.345534 24803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:24.354043 24803 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.66:38455
I20260926 16:42:24.354068 24931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.66:38455 every 8 connection(s)
I20260926 16:42:24.354862 24803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
I20260926 16:42:24.362020 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 24803
I20260926 16:42:24.362146 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance
I20260926 16:42:24.362474 21553 external_mini_cluster.cc:1428] Setting key a582c33cdee994716f0efe48d2ae6a17
Sep 26 16:42:24 dist-test-slave-n1hw krb5kdc[24562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440944, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.66@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:42:24.368930 24646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:24.356611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:33033 (local address 127.21.12.126:37663)
0926 16:42:24.356743 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:24.356746 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:24.357534 (+ 788us) server_negotiation.cc:407] Connection header received
0926 16:42:24.358975 (+ 1441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:24.358978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:24.359023 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:24.359101 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:24.360265 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:24.360801 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:24.361988 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:24.362369 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:24.362491 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:24.364955 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:24.364959 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:24.364961 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:24.364984 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:24.366683 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:24.367313 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:24.367315 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:24.367317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:24.367389 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:24.367842 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:24.367845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:24.367846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:24.368005 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:24.368077 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:24.368692 (+ 615us) server_negotiation.cc:299] Negotiation successful
0926 16:42:24.368804 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 16:42:24.370189 24934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:24.356967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:37663 (local address 127.21.12.66:33033)
0926 16:42:24.357385 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:24.357429 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:24.358741 (+ 1312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:24.359251 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:24.359257 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:24.359605 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:24.360050 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:24.360062 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:24.360912 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:24.360915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:24.361868 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:24.361877 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:24.362512 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:24.362534 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:24.362709 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:24.363345 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:24.363361 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:24.364776 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:24.366806 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:24.366815 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:24.366833 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:24.367208 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:24.367478 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:24.367484 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:24.367489 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:24.367676 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:24.368110 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:24.368117 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:24.368432 (+ 315us) client_negotiation.cc:769] Sending connection context
0926 16:42:24.368656 (+ 224us) client_negotiation.cc:240] Negotiation successful
0926 16:42:24.369044 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":102,"threads_started":1}
I20260926 16:42:24.371377 24932 heartbeater.cc:347] Connected to a master server at 127.21.12.126:37663
I20260926 16:42:24.371636 24932 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:24.372217 24932 heartbeater.cc:511] Master 127.21.12.126:37663 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.21.12.67@KRBTEST.COM; defaulting to no policy
I20260926 16:42:24.373420 24607 ts_manager.cc:194] Registered new tserver with Master: 920d365e25054fdcbde52bb31f57d6d1 (127.21.12.66:38455)
I20260926 16:42:24.374065 24607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.66@KRBTEST.COM'} at 127.21.12.66:33033
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab.
Entry for principal kudu/127.21.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab.
I20260926 16:42:24.390416 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.67:0
--local_ip_for_outbound_sockets=127.21.12.67
--webserver_interface=127.21.12.67
--webserver_port=0
--tserver_master_addrs=127.21.12.126:37663
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab
--principal=kudu/127.21.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:32825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:24.498558 24937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:24.498811 24937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:24.498854 24937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:24.503629 24937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:24.503682 24937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:24.503770 24937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.67
I20260926 16:42:24.509001 24937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:32825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab
--principal=kudu/127.21.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.67
--webserver_port=0
--tserver_master_addrs=127.21.12.126:37663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.67
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:24.510058 24937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:24.511307 24937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:24.520776 24943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:24.520814 24942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:24.520958 24937 server_base.cc:1034] running on GCE node
W20260926 16:42:24.520761 24945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:24.521464 24937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:24.522101 24937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:24.523283 24937 hybrid_clock.cc:648] HybridClock initialized: now 1790440944523268 us; error 62 us; skew 500 ppm
Sep 26 16:42:24 dist-test-slave-n1hw krb5kdc[24562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440944, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:24.526016 24937 init.cc:377] Logged in from keytab as kudu/127.21.12.67@KRBTEST.COM (short username kudu)
I20260926 16:42:24.526877 24937 webserver.cc:460] Webserver started at http://127.21.12.67:44299/ using document root <none> and password file <none>
I20260926 16:42:24.527412 24937 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:24.527499 24937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:24.527701 24937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:24.529601 24937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-2/data/instance:
uuid: "0457a76192584a2bb4d218f78a1857fe"
format_stamp: "Formatted at 2026-09-26 16:42:24 on dist-test-slave-n1hw"
server_key: "0e947caa0b1c58b7e384c46c29991c4f"
server_key_iv: "42c3e58c6344f3f929c06f6c1c771212"
server_key_version: "encryptionkey@0"
I20260926 16:42:24.530203 24937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance:
uuid: "0457a76192584a2bb4d218f78a1857fe"
format_stamp: "Formatted at 2026-09-26 16:42:24 on dist-test-slave-n1hw"
server_key: "0e947caa0b1c58b7e384c46c29991c4f"
server_key_iv: "42c3e58c6344f3f929c06f6c1c771212"
server_key_version: "encryptionkey@0"
I20260926 16:42:24.533576 24937 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:42:24.535866 24952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:24.536963 24937 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:42:24.537094 24937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-2/wal
uuid: "0457a76192584a2bb4d218f78a1857fe"
format_stamp: "Formatted at 2026-09-26 16:42:24 on dist-test-slave-n1hw"
server_key: "0e947caa0b1c58b7e384c46c29991c4f"
server_key_iv: "42c3e58c6344f3f929c06f6c1c771212"
server_key_version: "encryptionkey@0"
I20260926 16:42:24.537194 24937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:24.559850 24937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:24.563578 24937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:24.563797 24937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:24.564421 24937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:42:24.565261 24937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:24.565331 24937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:24.565400 24937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:24.565449 24937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:24.576718 24937 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.67:36301
I20260926 16:42:24.576773 25065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.67:36301 every 8 connection(s)
I20260926 16:42:24.578284 24937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
I20260926 16:42:24.586380 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 24937
I20260926 16:42:24.586519 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance
I20260926 16:42:24.586829 21553 external_mini_cluster.cc:1428] Setting key 24be56802136729dc9aeee4603b33665
Sep 26 16:42:24 dist-test-slave-n1hw krb5kdc[24562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440944, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.67@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:42:24.592895 24646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:24.580304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:57783 (local address 127.21.12.126:37663)
0926 16:42:24.580444 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:24.580448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:24.581238 (+ 790us) server_negotiation.cc:407] Connection header received
0926 16:42:24.582665 (+ 1427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:24.582668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:24.582713 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:24.582798 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:24.583978 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:24.584729 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:24.586065 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:24.586292 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:24.586381 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:24.589593 (+ 3212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:24.589596 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:24.589598 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:24.589622 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:24.591181 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:24.591632 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:24.591634 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:24.591635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:24.591708 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:24.591989 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:24.591991 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:24.591992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:24.592153 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:24.592227 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:24.592675 (+ 448us) server_negotiation.cc:299] Negotiation successful
0926 16:42:24.592770 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 16:42:24.593704 25068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:24.580650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:37663 (local address 127.21.12.67:57783)
0926 16:42:24.581085 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:24.581126 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:24.582455 (+ 1329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:24.582959 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:24.582965 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:24.583374 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:24.583850 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:24.583863 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:24.584883 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:24.584886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:24.585925 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:24.585936 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:24.586471 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:24.586477 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:24.586694 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:24.587641 (+ 947us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:24.587664 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:24.589412 (+ 1748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:24.591287 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:24.591291 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:24.591303 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:24.591548 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:24.591785 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:24.591787 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:24.591788 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:24.591913 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:24.592236 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:24.592240 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:24.592490 (+ 250us) client_negotiation.cc:769] Sending connection context
0926 16:42:24.592647 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 16:42:24.592890 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":105,"threads_started":1}
I20260926 16:42:24.594933 25066 heartbeater.cc:347] Connected to a master server at 127.21.12.126:37663
I20260926 16:42:24.595170 25066 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:24.595635 25066 heartbeater.cc:511] Master 127.21.12.126:37663 requested a full tablet report, sending...
I20260926 16:42:24.596719 24607 ts_manager.cc:194] Registered new tserver with Master: 0457a76192584a2bb4d218f78a1857fe (127.21.12.67:36301)
I20260926 16:42:24.597306 24607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.67@KRBTEST.COM'} at 127.21.12.67:57783
I20260926 16:42:24.600910 21553 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 16:42:24 dist-test-slave-n1hw krb5kdc[24562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440944, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 16:42:24 dist-test-slave-n1hw krb5kdc[24562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440944, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:42:24.631774 24646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:24.622734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59184 (local address 127.21.12.126:37663)
0926 16:42:24.622893 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:24.622900 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:24.623031 (+ 131us) server_negotiation.cc:407] Connection header received
0926 16:42:24.623145 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:24.623151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:24.623222 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:24.623337 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:24.624016 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:24.624913 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:24.625799 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:24.626054 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:24.626137 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:24.628611 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:24.628620 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:24.628622 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:24.628645 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:24.630132 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:24.630545 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:24.630550 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:24.630552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:24.630631 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:24.630909 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:24.630913 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:24.630914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:24.631121 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:24.631262 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:24.631500 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 16:42:24.631614 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 16:42:24.632131 24607 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:59184
I20260926 16:42:24.632977 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 24669
I20260926 16:42:24.638903 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 24803
I20260926 16:42:24.644459 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 24937
I20260926 16:42:24.649578 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 24575
2026-09-26T16:42:24Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2113 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:42:26 dist-test-slave-n1hw krb5kdc[25078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:42:26 dist-test-slave-n1hw krb5kdc[25078](info): set up 2 sockets
Sep 26 16:42:26 dist-test-slave-n1hw krb5kdc[25078](info): commencing operation
krb5kdc: starting...
W20260926 16:42:27.395460 21553 mini_kdc.cc:121] Time spent starting KDC: real 2.724s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260926 16:42:27.705533 21553 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.310s user 0.001s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260926 16:42:27.956418 21553 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.251s user 0.002s sys 0.000s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260926 16:42:28.185405 21553 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.229s user 0.002s sys 0.000s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/krb5kdc/test-user.keytab.
W20260926 16:42:28.570092 21553 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.385s user 0.002s sys 0.000s
Sep 26 16:42:28 dist-test-slave-n1hw krb5kdc[25078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:42:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:42:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
I20260926 16:42:28.614904 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:44767
--webserver_interface=127.21.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:44999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:44767
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:28.728637 25091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:28.728897 25091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:28.728937 25091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:28.732997 25091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:42:28.733049 25091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:28.733068 25091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:28.733085 25091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:42:28.733099 25091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:42:28.738343 25091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:44999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:44767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:44767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:28.739396 25091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:28.740460 25091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:28.748978 25097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:28.749029 25099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:28.749029 25096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:28.749127 25091 server_base.cc:1034] running on GCE node
I20260926 16:42:28.749744 25091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:28.750944 25091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:28.752104 25091 hybrid_clock.cc:648] HybridClock initialized: now 1790440948752077 us; error 45 us; skew 500 ppm
Sep 26 16:42:28 dist-test-slave-n1hw krb5kdc[25078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440948, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:28.760437 25091 init.cc:377] Logged in from keytab as kudu/127.21.12.126@KRBTEST.COM (short username kudu)
I20260926 16:42:28.761391 25091 webserver.cc:460] Webserver started at http://127.21.12.126:43305/ using document root <none> and password file <none>
I20260926 16:42:28.762084 25091 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:28.762177 25091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:28.762423 25091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:28.764492 25091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "3de9ac3c280244be8878aedccff08ee5"
format_stamp: "Formatted at 2026-09-26 16:42:28 on dist-test-slave-n1hw"
server_key: "1bf3b0a4d674cdaf011eaa38997d6d86"
server_key_iv: "e34bab6e6549a430f99d97b6408dfae5"
server_key_version: "encryptionkey@0"
I20260926 16:42:28.765071 25091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "3de9ac3c280244be8878aedccff08ee5"
format_stamp: "Formatted at 2026-09-26 16:42:28 on dist-test-slave-n1hw"
server_key: "1bf3b0a4d674cdaf011eaa38997d6d86"
server_key_iv: "e34bab6e6549a430f99d97b6408dfae5"
server_key_version: "encryptionkey@0"
I20260926 16:42:28.768819 25091 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 16:42:28.771377 25106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:28.772667 25091 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 16:42:28.772809 25091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "3de9ac3c280244be8878aedccff08ee5"
format_stamp: "Formatted at 2026-09-26 16:42:28 on dist-test-slave-n1hw"
server_key: "1bf3b0a4d674cdaf011eaa38997d6d86"
server_key_iv: "e34bab6e6549a430f99d97b6408dfae5"
server_key_version: "encryptionkey@0"
I20260926 16:42:28.772925 25091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:28.784004 25091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:28.787451 25091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:28.787662 25091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:28.795051 25091 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.126:44767
I20260926 16:42:28.795066 25158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.126:44767 every 8 connection(s)
I20260926 16:42:28.796021 25091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:42:28.799137 25159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:28.800477 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 25091
I20260926 16:42:28.800616 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:42:28.800963 21553 external_mini_cluster.cc:1428] Setting key 31d99a8efc5ee7852b348012b35747ac
I20260926 16:42:28.806396 25159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3de9ac3c280244be8878aedccff08ee5: Bootstrap starting.
I20260926 16:42:28.809369 25159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3de9ac3c280244be8878aedccff08ee5: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:28.810365 25159 log.cc:824] T 00000000000000000000000000000000 P 3de9ac3c280244be8878aedccff08ee5: Log is configured to *not* fsync() on all Append() calls
Sep 26 16:42:28 dist-test-slave-n1hw krb5kdc[25078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:42:28.812849 25159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3de9ac3c280244be8878aedccff08ee5: No bootstrap required, opened a new log
I20260926 16:42:28.816391 25162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:28.802466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38602 (local address 127.21.12.126:44767)
0926 16:42:28.802857 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:28.802882 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:28.802938 (+ 56us) server_negotiation.cc:407] Connection header received
0926 16:42:28.803566 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:28.803583 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:28.803931 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:28.804212 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:28.805191 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:28.806247 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:28.807478 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:28.807961 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:28.808098 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:28.811188 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:28.811193 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:28.811241 (+ 48us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:28.811323 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:28.813293 (+ 1970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:28.813714 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:28.813720 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:28.813726 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:28.813893 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:28.814144 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:28.814147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:28.814148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:28.814655 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:28.814850 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:28.815070 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 16:42:28.815428 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":125,"threads_started":1}
I20260926 16:42:28.817380 25159 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3de9ac3c280244be8878aedccff08ee5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de9ac3c280244be8878aedccff08ee5" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 44767 } }
I20260926 16:42:28.817625 25159 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3de9ac3c280244be8878aedccff08ee5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:28.817701 25159 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3de9ac3c280244be8878aedccff08ee5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3de9ac3c280244be8878aedccff08ee5, State: Initialized, Role: FOLLOWER
I20260926 16:42:28.818186 25159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3de9ac3c280244be8878aedccff08ee5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de9ac3c280244be8878aedccff08ee5" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 44767 } }
I20260926 16:42:28.818311 25159 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3de9ac3c280244be8878aedccff08ee5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:42:28.818397 25159 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3de9ac3c280244be8878aedccff08ee5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:42:28.818495 25159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3de9ac3c280244be8878aedccff08ee5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:28.819372 25159 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3de9ac3c280244be8878aedccff08ee5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de9ac3c280244be8878aedccff08ee5" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 44767 } }
I20260926 16:42:28.819669 25159 leader_election.cc:302] T 00000000000000000000000000000000 P 3de9ac3c280244be8878aedccff08ee5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3de9ac3c280244be8878aedccff08ee5; no voters:
I20260926 16:42:28.819916 25159 leader_election.cc:288] T 00000000000000000000000000000000 P 3de9ac3c280244be8878aedccff08ee5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:42:28.820063 25164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3de9ac3c280244be8878aedccff08ee5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:28.820286 25164 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3de9ac3c280244be8878aedccff08ee5 [term 1 LEADER]: Becoming Leader. State: Replica: 3de9ac3c280244be8878aedccff08ee5, State: Running, Role: LEADER
I20260926 16:42:28.820631 25164 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3de9ac3c280244be8878aedccff08ee5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de9ac3c280244be8878aedccff08ee5" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 44767 } }
I20260926 16:42:28.821143 25159 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3de9ac3c280244be8878aedccff08ee5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:42:28.822465 25166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3de9ac3c280244be8878aedccff08ee5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3de9ac3c280244be8878aedccff08ee5. Latest consensus state: current_term: 1 leader_uuid: "3de9ac3c280244be8878aedccff08ee5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de9ac3c280244be8878aedccff08ee5" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 44767 } } }
I20260926 16:42:28.822479 25165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3de9ac3c280244be8878aedccff08ee5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3de9ac3c280244be8878aedccff08ee5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de9ac3c280244be8878aedccff08ee5" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 44767 } } }
I20260926 16:42:28.822614 25165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3de9ac3c280244be8878aedccff08ee5 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:28.822613 25166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3de9ac3c280244be8878aedccff08ee5 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:28.823618 25173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:42:28.826800 25173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:42:28.831670 25173 catalog_manager.cc:1347] Generated new cluster ID: 646396ee52144e05b89e9c3df0e7dcdb
I20260926 16:42:28.831725 25173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:42:28.844208 25173 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:42:28.845372 25173 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:42:28.857508 25173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3de9ac3c280244be8878aedccff08ee5: Generated new TSK 0
I20260926 16:42:28.858294 25173 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab.
Entry for principal kudu/127.21.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab.
I20260926 16:42:28.887019 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.65:0
--local_ip_for_outbound_sockets=127.21.12.65
--webserver_interface=127.21.12.65
--webserver_port=0
--tserver_master_addrs=127.21.12.126:44767
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab
--principal=kudu/127.21.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:44999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:28.996694 25185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:28.996940 25185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:28.997014 25185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:29.001073 25185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:29.001132 25185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:29.001223 25185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.65
I20260926 16:42:29.006443 25185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:44999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab
--principal=kudu/127.21.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.65
--webserver_port=0
--tserver_master_addrs=127.21.12.126:44767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.65
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:29.007426 25185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:29.008424 25185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:29.017966 25191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:29.017973 25193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:29.018002 25190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:29.018062 25185 server_base.cc:1034] running on GCE node
I20260926 16:42:29.018591 25185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:29.019207 25185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:29.020443 25185 hybrid_clock.cc:648] HybridClock initialized: now 1790440949020405 us; error 51 us; skew 500 ppm
Sep 26 16:42:29 dist-test-slave-n1hw krb5kdc[25078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440949, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:29.023459 25185 init.cc:377] Logged in from keytab as kudu/127.21.12.65@KRBTEST.COM (short username kudu)
I20260926 16:42:29.024232 25185 webserver.cc:460] Webserver started at http://127.21.12.65:43801/ using document root <none> and password file <none>
I20260926 16:42:29.024787 25185 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:29.024876 25185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:29.025094 25185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:29.027091 25185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-0/data/instance:
uuid: "f1990f9631f34d07af50a59dbfc5c90d"
format_stamp: "Formatted at 2026-09-26 16:42:29 on dist-test-slave-n1hw"
server_key: "ef55428825e8de15a2138b77c988614e"
server_key_iv: "7725a189fb173fe0dfd35b39c9f7f92a"
server_key_version: "encryptionkey@0"
I20260926 16:42:29.027643 25185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance:
uuid: "f1990f9631f34d07af50a59dbfc5c90d"
format_stamp: "Formatted at 2026-09-26 16:42:29 on dist-test-slave-n1hw"
server_key: "ef55428825e8de15a2138b77c988614e"
server_key_iv: "7725a189fb173fe0dfd35b39c9f7f92a"
server_key_version: "encryptionkey@0"
I20260926 16:42:29.031222 25185 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:42:29.033623 25200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:29.034868 25185 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:42:29.035005 25185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-0/wal
uuid: "f1990f9631f34d07af50a59dbfc5c90d"
format_stamp: "Formatted at 2026-09-26 16:42:29 on dist-test-slave-n1hw"
server_key: "ef55428825e8de15a2138b77c988614e"
server_key_iv: "7725a189fb173fe0dfd35b39c9f7f92a"
server_key_version: "encryptionkey@0"
I20260926 16:42:29.035115 25185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:29.058658 25185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:29.061789 25185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:29.062007 25185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:29.062605 25185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:42:29.063457 25185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:29.063504 25185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:29.063544 25185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:29.063558 25185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:29.072710 25185 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.65:36207
I20260926 16:42:29.072875 25313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.65:36207 every 8 connection(s)
I20260926 16:42:29.073668 25185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
I20260926 16:42:29.083132 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 25185
I20260926 16:42:29.083235 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance
I20260926 16:42:29.083536 21553 external_mini_cluster.cc:1428] Setting key c57f68a20fc2f43f8839a15de3a24b64
Sep 26 16:42:29 dist-test-slave-n1hw krb5kdc[25078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440949, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:42:29.088774 25162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:29.075726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:44225 (local address 127.21.12.126:44767)
0926 16:42:29.075917 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:29.075922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:29.076765 (+ 843us) server_negotiation.cc:407] Connection header received
0926 16:42:29.078405 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:29.078408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:29.078454 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:29.078527 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:29.079832 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:29.080356 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:29.081515 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:29.081728 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:29.081864 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:29.084648 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:29.084652 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:29.084654 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:29.084702 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:29.086579 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:29.087081 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:29.087084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:29.087087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:29.087192 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:29.087540 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:29.087543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:29.087549 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:29.087792 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:29.087879 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:29.088349 (+ 470us) server_negotiation.cc:299] Negotiation successful
0926 16:42:29.088495 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 16:42:29.089367 25316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:29.076119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:44767 (local address 127.21.12.65:44225)
0926 16:42:29.076603 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:29.076648 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:29.078193 (+ 1545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:29.078699 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:29.078706 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:29.079097 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:29.079682 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:29.079698 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:29.080467 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:29.080470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:29.081407 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:29.081416 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:29.081914 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:29.081917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:29.082066 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:29.082723 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:29.082740 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:29.084457 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:29.086716 (+ 2259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:29.086720 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:29.086731 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:29.086980 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:29.087291 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:29.087293 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:29.087294 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:29.087441 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:29.087891 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:29.087895 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:29.088111 (+ 216us) client_negotiation.cc:769] Sending connection context
0926 16:42:29.088265 (+ 154us) client_negotiation.cc:240] Negotiation successful
0926 16:42:29.088511 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":98,"threads_started":1}
I20260926 16:42:29.090590 25314 heartbeater.cc:347] Connected to a master server at 127.21.12.126:44767
I20260926 16:42:29.090876 25314 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:29.091508 25314 heartbeater.cc:511] Master 127.21.12.126:44767 requested a full tablet report, sending...
I20260926 16:42:29.093120 25123 ts_manager.cc:194] Registered new tserver with Master: f1990f9631f34d07af50a59dbfc5c90d (127.21.12.65:36207)
WARNING: no policy specified for kudu/127.21.12.66@KRBTEST.COM; defaulting to no policy
I20260926 16:42:29.094363 25123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.65@KRBTEST.COM'} at 127.21.12.65:44225
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab.
Entry for principal kudu/127.21.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab.
I20260926 16:42:29.112715 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.66:0
--local_ip_for_outbound_sockets=127.21.12.66
--webserver_interface=127.21.12.66
--webserver_port=0
--tserver_master_addrs=127.21.12.126:44767
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab
--principal=kudu/127.21.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:44999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:29.222119 25319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:29.222375 25319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:29.222417 25319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:29.226588 25319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:29.226640 25319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:29.226697 25319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.66
I20260926 16:42:29.231742 25319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:44999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab
--principal=kudu/127.21.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.66
--webserver_port=0
--tserver_master_addrs=127.21.12.126:44767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.66
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:29.232676 25319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:29.233690 25319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:29.243142 25325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:29.243129 25324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:29.243237 25319 server_base.cc:1034] running on GCE node
W20260926 16:42:29.243132 25327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:29.243731 25319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:29.244396 25319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:29.245586 25319 hybrid_clock.cc:648] HybridClock initialized: now 1790440949245570 us; error 27 us; skew 500 ppm
Sep 26 16:42:29 dist-test-slave-n1hw krb5kdc[25078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440949, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:29.248359 25319 init.cc:377] Logged in from keytab as kudu/127.21.12.66@KRBTEST.COM (short username kudu)
I20260926 16:42:29.249218 25319 webserver.cc:460] Webserver started at http://127.21.12.66:44089/ using document root <none> and password file <none>
I20260926 16:42:29.249702 25319 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:29.249826 25319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:29.250015 25319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:29.251917 25319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-1/data/instance:
uuid: "d9d69de3c9604e6590363e94ee290ece"
format_stamp: "Formatted at 2026-09-26 16:42:29 on dist-test-slave-n1hw"
server_key: "8d29c3d43e71cee85ee87846b428be3d"
server_key_iv: "03a76df83a63543c9a7847ae5ce5de62"
server_key_version: "encryptionkey@0"
I20260926 16:42:29.252406 25319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance:
uuid: "d9d69de3c9604e6590363e94ee290ece"
format_stamp: "Formatted at 2026-09-26 16:42:29 on dist-test-slave-n1hw"
server_key: "8d29c3d43e71cee85ee87846b428be3d"
server_key_iv: "03a76df83a63543c9a7847ae5ce5de62"
server_key_version: "encryptionkey@0"
I20260926 16:42:29.255859 25319 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 16:42:29.258090 25334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:29.259229 25319 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 16:42:29.259321 25319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-1/wal
uuid: "d9d69de3c9604e6590363e94ee290ece"
format_stamp: "Formatted at 2026-09-26 16:42:29 on dist-test-slave-n1hw"
server_key: "8d29c3d43e71cee85ee87846b428be3d"
server_key_iv: "03a76df83a63543c9a7847ae5ce5de62"
server_key_version: "encryptionkey@0"
I20260926 16:42:29.259410 25319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:29.272467 25319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:29.275519 25319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:29.275657 25319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:29.276120 25319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:42:29.276916 25319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:29.276961 25319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:29.276998 25319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:29.277020 25319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:29.285898 25319 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.66:42315
I20260926 16:42:29.285925 25447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.66:42315 every 8 connection(s)
I20260926 16:42:29.286715 25319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
I20260926 16:42:29.288223 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 25319
I20260926 16:42:29.288362 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance
I20260926 16:42:29.288679 21553 external_mini_cluster.cc:1428] Setting key a703e9fe145be4c274c2526c9e029417
Sep 26 16:42:29 dist-test-slave-n1hw krb5kdc[25078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440949, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.66@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.12.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.67@KRBTEST.COM" created.
I20260926 16:42:29.301188 25162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:29.288428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:42031 (local address 127.21.12.126:44767)
0926 16:42:29.288567 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:29.288571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:29.289402 (+ 831us) server_negotiation.cc:407] Connection header received
0926 16:42:29.290926 (+ 1524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:29.290932 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:29.290991 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:29.291084 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:29.292178 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:29.292701 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:29.294508 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:29.294729 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:29.294831 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:29.297553 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:29.297559 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:29.297566 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:29.297589 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:29.299057 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:29.299595 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:29.299600 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:29.299602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:29.299683 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:29.300030 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:29.300033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:29.300035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:29.300217 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:29.300336 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:29.300895 (+ 559us) server_negotiation.cc:299] Negotiation successful
0926 16:42:29.301020 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 16:42:29.302049 25450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:29.288783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:44767 (local address 127.21.12.66:42031)
0926 16:42:29.289252 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:29.289301 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:29.290720 (+ 1419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:29.291233 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:29.291240 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:29.291591 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:29.292032 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:29.292045 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:29.292830 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:29.292836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:29.294357 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:29.294371 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:29.294894 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:29.294898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:29.295128 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:29.295715 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:29.295741 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:29.297391 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:29.299216 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:29.299222 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:29.299235 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:29.299487 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:29.299792 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:29.299795 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:29.299797 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:29.299930 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:29.300334 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:29.300341 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:29.300631 (+ 290us) client_negotiation.cc:769] Sending connection context
0926 16:42:29.300856 (+ 225us) client_negotiation.cc:240] Negotiation successful
0926 16:42:29.301161 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":143,"threads_started":1}
I20260926 16:42:29.303380 25448 heartbeater.cc:347] Connected to a master server at 127.21.12.126:44767
I20260926 16:42:29.303628 25448 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:29.304103 25448 heartbeater.cc:511] Master 127.21.12.126:44767 requested a full tablet report, sending...
I20260926 16:42:29.305188 25123 ts_manager.cc:194] Registered new tserver with Master: d9d69de3c9604e6590363e94ee290ece (127.21.12.66:42315)
I20260926 16:42:29.306087 25123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.66@KRBTEST.COM'} at 127.21.12.66:42031
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab.
Entry for principal kudu/127.21.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab.
I20260926 16:42:29.316901 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.67:0
--local_ip_for_outbound_sockets=127.21.12.67
--webserver_interface=127.21.12.67
--webserver_port=0
--tserver_master_addrs=127.21.12.126:44767
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab
--principal=kudu/127.21.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:44999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:29.423805 25453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:29.424041 25453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:29.424109 25453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:29.428244 25453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:29.428340 25453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:29.428452 25453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.67
I20260926 16:42:29.433570 25453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:44999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab
--principal=kudu/127.21.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.67
--webserver_port=0
--tserver_master_addrs=127.21.12.126:44767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.67
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:29.434576 25453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:29.435559 25453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:29.445176 25458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:29.445214 25461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:29.445204 25459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:29.445492 25453 server_base.cc:1034] running on GCE node
I20260926 16:42:29.445920 25453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:29.446523 25453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:29.447715 25453 hybrid_clock.cc:648] HybridClock initialized: now 1790440949447689 us; error 47 us; skew 500 ppm
Sep 26 16:42:29 dist-test-slave-n1hw krb5kdc[25078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440949, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:29.450420 25453 init.cc:377] Logged in from keytab as kudu/127.21.12.67@KRBTEST.COM (short username kudu)
I20260926 16:42:29.451303 25453 webserver.cc:460] Webserver started at http://127.21.12.67:38923/ using document root <none> and password file <none>
I20260926 16:42:29.451840 25453 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:29.451929 25453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:29.452150 25453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:29.454126 25453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-2/data/instance:
uuid: "69fdf4662e564af9808e258fd707a656"
format_stamp: "Formatted at 2026-09-26 16:42:29 on dist-test-slave-n1hw"
server_key: "02305be1acd98335bdd067773ae85e1a"
server_key_iv: "28d8f8496381957e610c71958627381d"
server_key_version: "encryptionkey@0"
I20260926 16:42:29.454679 25453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance:
uuid: "69fdf4662e564af9808e258fd707a656"
format_stamp: "Formatted at 2026-09-26 16:42:29 on dist-test-slave-n1hw"
server_key: "02305be1acd98335bdd067773ae85e1a"
server_key_iv: "28d8f8496381957e610c71958627381d"
server_key_version: "encryptionkey@0"
I20260926 16:42:29.458148 25453 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:42:29.460364 25468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:29.461514 25453 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:42:29.461653 25453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-2/wal
uuid: "69fdf4662e564af9808e258fd707a656"
format_stamp: "Formatted at 2026-09-26 16:42:29 on dist-test-slave-n1hw"
server_key: "02305be1acd98335bdd067773ae85e1a"
server_key_iv: "28d8f8496381957e610c71958627381d"
server_key_version: "encryptionkey@0"
I20260926 16:42:29.461758 25453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:29.474190 25453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:29.477068 25453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:29.477252 25453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:29.477839 25453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:42:29.478605 25453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:29.478674 25453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:29.478736 25453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:29.478796 25453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:29.487655 25453 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.67:38793
I20260926 16:42:29.487684 25581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.67:38793 every 8 connection(s)
I20260926 16:42:29.488485 25453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
I20260926 16:42:29.492456 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 25453
I20260926 16:42:29.492584 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance
I20260926 16:42:29.492905 21553 external_mini_cluster.cc:1428] Setting key 281a71cb86f3a91f97fa4d5d10c27430
Sep 26 16:42:29 dist-test-slave-n1hw krb5kdc[25078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440949, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.67@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:42:29.502424 25162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:29.490311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:60419 (local address 127.21.12.126:44767)
0926 16:42:29.490449 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:29.490453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:29.491298 (+ 845us) server_negotiation.cc:407] Connection header received
0926 16:42:29.492921 (+ 1623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:29.492924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:29.493015 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:29.493100 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:29.494455 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:29.495067 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:29.496117 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:29.496328 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:29.496405 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:29.498981 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:29.498984 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:29.498986 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:29.499010 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:29.500229 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:29.500735 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:29.500740 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:29.500742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:29.500826 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:29.501177 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:29.501182 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:29.501183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:29.501363 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:29.501602 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:29.502138 (+ 536us) server_negotiation.cc:299] Negotiation successful
0926 16:42:29.502261 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 16:42:29.503247 25584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:29.490695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:44767 (local address 127.21.12.67:60419)
0926 16:42:29.491137 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:29.491177 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:29.492675 (+ 1498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:29.493293 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:29.493300 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:29.493699 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:29.494298 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:29.494312 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:29.495175 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:29.495177 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:29.496019 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:29.496027 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:29.496497 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:29.496501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:29.496646 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:29.497253 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:29.497269 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:29.498808 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:29.500347 (+ 1539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:29.500354 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:29.500368 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:29.500630 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:29.500927 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:29.500931 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:29.500933 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:29.501081 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:29.501521 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:29.501527 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:29.501895 (+ 368us) client_negotiation.cc:769] Sending connection context
0926 16:42:29.502093 (+ 198us) client_negotiation.cc:240] Negotiation successful
0926 16:42:29.502472 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":99,"threads_started":1}
I20260926 16:42:29.504393 25582 heartbeater.cc:347] Connected to a master server at 127.21.12.126:44767
I20260926 16:42:29.504612 25582 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:29.505044 25582 heartbeater.cc:511] Master 127.21.12.126:44767 requested a full tablet report, sending...
I20260926 16:42:29.506284 25123 ts_manager.cc:194] Registered new tserver with Master: 69fdf4662e564af9808e258fd707a656 (127.21.12.67:38793)
I20260926 16:42:29.506927 21553 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:42:29.506958 25123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.67@KRBTEST.COM'} at 127.21.12.67:60419
Sep 26 16:42:29 dist-test-slave-n1hw krb5kdc[25078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 16:42:29 dist-test-slave-n1hw krb5kdc[25078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.65@KRBTEST.COM
I20260926 16:42:29.533187 25589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:29.523692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47682 (local address 127.21.12.65:36207)
0926 16:42:29.524041 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:29.524050 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:29.524072 (+ 22us) server_negotiation.cc:407] Connection header received
0926 16:42:29.524119 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:29.524124 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:29.524346 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:29.524464 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:29.525099 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:29.525988 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:29.526801 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:29.527166 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:29.527281 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:29.529654 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:29.529662 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:29.529673 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:29.529696 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:29.531435 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:29.531885 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:29.531890 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:29.531891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:29.531968 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:29.532249 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:29.532253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:29.532254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:29.532474 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:29.532678 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:29.532866 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 16:42:29.533012 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":126,"threads_started":1}
Sep 26 16:42:29 dist-test-slave-n1hw krb5kdc[25078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:42:29.543743 25162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:29.535937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55870 (local address 127.21.12.126:44767)
0926 16:42:29.536074 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:29.536077 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:29.536148 (+ 71us) server_negotiation.cc:407] Connection header received
0926 16:42:29.536235 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:29.536238 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:29.536283 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:29.536354 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:29.536873 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:29.537373 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:29.538245 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:29.538613 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:29.538698 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:29.540983 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:29.540986 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:29.540988 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:29.541009 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:29.542305 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:29.542671 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:29.542673 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:29.542674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:29.542744 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:29.542978 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:29.542980 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:29.542981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:29.543170 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:29.543266 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:29.543446 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 16:42:29.543535 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260926 16:42:29.544160 25123 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55870
I20260926 16:42:29.544721 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 25185
I20260926 16:42:29.550205 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 25319
I20260926 16:42:29.555408 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 25453
I20260926 16:42:29.560258 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 25091
2026-09-26T16:42:29Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4909 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:42:29 dist-test-slave-n1hw krb5kdc[25594](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:42:29 dist-test-slave-n1hw krb5kdc[25594](info): set up 2 sockets
Sep 26 16:42:29 dist-test-slave-n1hw krb5kdc[25594](info): commencing operation
krb5kdc: starting...
W20260926 16:42:30.595948 21553 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Sep 26 16:42:30 dist-test-slave-n1hw krb5kdc[25594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:42:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:42:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
I20260926 16:42:30.710839 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:43167
--webserver_interface=127.21.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:39549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:43167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:30.818694 25607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:30.818924 25607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:30.818964 25607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:30.822943 25607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:42:30.822992 25607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:30.823011 25607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:30.823027 25607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:42:30.823042 25607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:42:30.828604 25607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:39549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:43167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:43167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:30.829586 25607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:30.830569 25607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:30.838541 25612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:30.838699 25613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:30.838896 25607 server_base.cc:1034] running on GCE node
W20260926 16:42:30.838531 25615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:30.839427 25607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:30.840545 25607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:30.841730 25607 hybrid_clock.cc:648] HybridClock initialized: now 1790440950841698 us; error 48 us; skew 500 ppm
Sep 26 16:42:30 dist-test-slave-n1hw krb5kdc[25594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440950, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:30.844488 25607 init.cc:377] Logged in from keytab as kudu/127.21.12.126@KRBTEST.COM (short username kudu)
I20260926 16:42:30.845239 25607 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:30.845300 25607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:30.845528 25607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:30.847589 25607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "493fe9e446a846958b679d0832e6b476"
format_stamp: "Formatted at 2026-09-26 16:42:30 on dist-test-slave-n1hw"
server_key: "d61ff969adfda17f77381f4f0295a606"
server_key_iv: "795f7a819e6fd9ef27e6a93c312d6f5f"
server_key_version: "encryptionkey@0"
I20260926 16:42:30.848155 25607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "493fe9e446a846958b679d0832e6b476"
format_stamp: "Formatted at 2026-09-26 16:42:30 on dist-test-slave-n1hw"
server_key: "d61ff969adfda17f77381f4f0295a606"
server_key_iv: "795f7a819e6fd9ef27e6a93c312d6f5f"
server_key_version: "encryptionkey@0"
I20260926 16:42:30.851703 25607 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 16:42:30.854017 25621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:30.855149 25607 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 16:42:30.855270 25607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "493fe9e446a846958b679d0832e6b476"
format_stamp: "Formatted at 2026-09-26 16:42:30 on dist-test-slave-n1hw"
server_key: "d61ff969adfda17f77381f4f0295a606"
server_key_iv: "795f7a819e6fd9ef27e6a93c312d6f5f"
server_key_version: "encryptionkey@0"
I20260926 16:42:30.855368 25607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:30.865051 25607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:30.867687 25607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:30.867877 25607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:30.874902 25607 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.126:43167
I20260926 16:42:30.875159 25673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.126:43167 every 8 connection(s)
I20260926 16:42:30.875533 25607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:42:30.876140 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 25607
I20260926 16:42:30.876264 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:42:30.876583 21553 external_mini_cluster.cc:1428] Setting key fc35d34387d78b555d12356528bf8c2c
I20260926 16:42:30.878607 25674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:30.884891 25674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 493fe9e446a846958b679d0832e6b476: Bootstrap starting.
Sep 26 16:42:30 dist-test-slave-n1hw krb5kdc[25594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:42:30.887096 25674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 493fe9e446a846958b679d0832e6b476: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:30.887790 25674 log.cc:824] T 00000000000000000000000000000000 P 493fe9e446a846958b679d0832e6b476: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:30.889879 25674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 493fe9e446a846958b679d0832e6b476: No bootstrap required, opened a new log
I20260926 16:42:30.890914 25677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:30.877873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58798 (local address 127.21.12.126:43167)
0926 16:42:30.878248 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:30.878273 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:30.878305 (+ 32us) server_negotiation.cc:407] Connection header received
0926 16:42:30.878989 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:30.879006 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:30.879310 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:30.879581 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:30.880396 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:30.881441 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:30.882593 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:30.883038 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:30.883151 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:30.885976 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:30.885982 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:30.885994 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:30.886021 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:30.887794 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:30.888245 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:30.888256 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:30.888261 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:30.888368 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:30.888657 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:30.888662 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:30.888663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:30.889133 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:30.889336 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:30.889674 (+ 338us) server_negotiation.cc:299] Negotiation successful
0926 16:42:30.889984 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":131,"threads_started":1}
I20260926 16:42:30.893466 25674 raft_consensus.cc:348] T 00000000000000000000000000000000 P 493fe9e446a846958b679d0832e6b476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "493fe9e446a846958b679d0832e6b476" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 43167 } }
I20260926 16:42:30.893671 25674 raft_consensus.cc:374] T 00000000000000000000000000000000 P 493fe9e446a846958b679d0832e6b476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:30.893704 25674 raft_consensus.cc:729] T 00000000000000000000000000000000 P 493fe9e446a846958b679d0832e6b476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 493fe9e446a846958b679d0832e6b476, State: Initialized, Role: FOLLOWER
I20260926 16:42:30.894110 25674 consensus_queue.cc:260] T 00000000000000000000000000000000 P 493fe9e446a846958b679d0832e6b476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "493fe9e446a846958b679d0832e6b476" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 43167 } }
I20260926 16:42:30.894224 25674 raft_consensus.cc:388] T 00000000000000000000000000000000 P 493fe9e446a846958b679d0832e6b476 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:42:30.894271 25674 raft_consensus.cc:482] T 00000000000000000000000000000000 P 493fe9e446a846958b679d0832e6b476 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:42:30.894337 25674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 493fe9e446a846958b679d0832e6b476 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:30.895119 25674 raft_consensus.cc:504] T 00000000000000000000000000000000 P 493fe9e446a846958b679d0832e6b476 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "493fe9e446a846958b679d0832e6b476" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 43167 } }
I20260926 16:42:30.895382 25674 leader_election.cc:302] T 00000000000000000000000000000000 P 493fe9e446a846958b679d0832e6b476 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 493fe9e446a846958b679d0832e6b476; no voters:
I20260926 16:42:30.895612 25674 leader_election.cc:288] T 00000000000000000000000000000000 P 493fe9e446a846958b679d0832e6b476 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:42:30.895759 25679 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 493fe9e446a846958b679d0832e6b476 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:30.896005 25679 raft_consensus.cc:686] T 00000000000000000000000000000000 P 493fe9e446a846958b679d0832e6b476 [term 1 LEADER]: Becoming Leader. State: Replica: 493fe9e446a846958b679d0832e6b476, State: Running, Role: LEADER
I20260926 16:42:30.896346 25679 consensus_queue.cc:237] T 00000000000000000000000000000000 P 493fe9e446a846958b679d0832e6b476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "493fe9e446a846958b679d0832e6b476" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 43167 } }
I20260926 16:42:30.896786 25674 sys_catalog.cc:564] T 00000000000000000000000000000000 P 493fe9e446a846958b679d0832e6b476 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:42:30.898597 25680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 493fe9e446a846958b679d0832e6b476 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "493fe9e446a846958b679d0832e6b476" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "493fe9e446a846958b679d0832e6b476" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 43167 } } }
I20260926 16:42:30.898734 25680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 493fe9e446a846958b679d0832e6b476 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:30.898936 25681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 493fe9e446a846958b679d0832e6b476 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 493fe9e446a846958b679d0832e6b476. Latest consensus state: current_term: 1 leader_uuid: "493fe9e446a846958b679d0832e6b476" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "493fe9e446a846958b679d0832e6b476" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 43167 } } }
I20260926 16:42:30.899024 25681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 493fe9e446a846958b679d0832e6b476 [sys.catalog]: This master's current role is: LEADER
W20260926 16:42:30.901844 25695 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 493fe9e446a846958b679d0832e6b476: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 16:42:30.901945 25695 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 16:42:30.902050 25694 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:42:30.902942 25694 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:42:30.908068 25694 catalog_manager.cc:1347] Generated new cluster ID: cd1a6b6c912d487e90d0cbca0a2cb7ab
I20260926 16:42:30.908129 25694 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:42:30.917368 25694 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:42:30.918473 25694 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:42:30.924630 25694 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 493fe9e446a846958b679d0832e6b476: Generated new TSK 0
I20260926 16:42:30.925194 25694 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab.
Entry for principal kudu/127.21.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab.
I20260926 16:42:30.956159 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.65:0
--local_ip_for_outbound_sockets=127.21.12.65
--webserver_interface=127.21.12.65
--webserver_port=0
--tserver_master_addrs=127.21.12.126:43167
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab
--principal=kudu/127.21.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:39549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:31.103272 25700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:31.103526 25700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:31.103570 25700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:31.107566 25700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:31.107620 25700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:31.107712 25700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.65
I20260926 16:42:31.112768 25700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:39549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab
--principal=kudu/127.21.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.65
--webserver_port=0
--tserver_master_addrs=127.21.12.126:43167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.65
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:31.113732 25700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:31.114784 25700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 16:42:31.123939 25700 server_base.cc:1034] running on GCE node
W20260926 16:42:31.123965 25706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:31.124029 25705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:31.123980 25708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:31.124517 25700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:31.125118 25700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:31.126387 25700 hybrid_clock.cc:648] HybridClock initialized: now 1790440951126330 us; error 32 us; skew 500 ppm
Sep 26 16:42:31 dist-test-slave-n1hw krb5kdc[25594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440951, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:31.129196 25700 init.cc:377] Logged in from keytab as kudu/127.21.12.65@KRBTEST.COM (short username kudu)
I20260926 16:42:31.130007 25700 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:31.130095 25700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:31.130314 25700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:31.132261 25700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-0/data/instance:
uuid: "bd493bcb747c47609b20e8ac0ab6864b"
format_stamp: "Formatted at 2026-09-26 16:42:31 on dist-test-slave-n1hw"
server_key: "97d8a362104a40522af320d133b7660a"
server_key_iv: "7a0aaa54fce4a3279b07d284995a0377"
server_key_version: "encryptionkey@0"
I20260926 16:42:31.132810 25700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance:
uuid: "bd493bcb747c47609b20e8ac0ab6864b"
format_stamp: "Formatted at 2026-09-26 16:42:31 on dist-test-slave-n1hw"
server_key: "97d8a362104a40522af320d133b7660a"
server_key_iv: "7a0aaa54fce4a3279b07d284995a0377"
server_key_version: "encryptionkey@0"
I20260926 16:42:31.136626 25700 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 16:42:31.138929 25714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:31.140029 25700 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 16:42:31.140161 25700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-0/wal
uuid: "bd493bcb747c47609b20e8ac0ab6864b"
format_stamp: "Formatted at 2026-09-26 16:42:31 on dist-test-slave-n1hw"
server_key: "97d8a362104a40522af320d133b7660a"
server_key_iv: "7a0aaa54fce4a3279b07d284995a0377"
server_key_version: "encryptionkey@0"
I20260926 16:42:31.140265 25700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:31.154000 25700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:31.156788 25700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:31.156989 25700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:31.157565 25700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:42:31.158522 25700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:31.158595 25700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:31.158679 25700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:31.158740 25700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:31.168049 25700 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.65:41435
I20260926 16:42:31.168080 25827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.65:41435 every 8 connection(s)
I20260926 16:42:31.168613 25700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
I20260926 16:42:31.172956 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 25700
I20260926 16:42:31.173095 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance
I20260926 16:42:31.173365 21553 external_mini_cluster.cc:1428] Setting key bdf289483a606a7800d90afb199d4c20
Sep 26 16:42:31 dist-test-slave-n1hw krb5kdc[25594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440951, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:42:31.182791 25677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:31.170239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:50743 (local address 127.21.12.126:43167)
0926 16:42:31.170378 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:31.170382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:31.171182 (+ 800us) server_negotiation.cc:407] Connection header received
0926 16:42:31.172460 (+ 1278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:31.172463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:31.172507 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:31.172579 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:31.173989 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:31.174601 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:31.175858 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:31.176273 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:31.176390 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:31.179230 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:31.179234 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:31.179236 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:31.179257 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:31.180738 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:31.181248 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:31.181253 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:31.181255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:31.181334 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:31.181666 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:31.181671 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:31.181673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:31.181900 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:31.182046 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:31.182505 (+ 459us) server_negotiation.cc:299] Negotiation successful
0926 16:42:31.182625 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
WARNING: no policy specified for kudu/127.21.12.66@KRBTEST.COM; defaulting to no policy
I20260926 16:42:31.183528 25830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:31.170587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43167 (local address 127.21.12.65:50743)
0926 16:42:31.171028 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:31.171070 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:31.172257 (+ 1187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:31.172748 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:31.172755 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:31.173194 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:31.173811 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:31.173832 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:31.174730 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:31.174734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:31.175724 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:31.175734 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:31.176459 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:31.176465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:31.176696 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:31.177292 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:31.177322 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:31.179073 (+ 1751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:31.180870 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:31.180876 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:31.180889 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:31.181145 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:31.181425 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:31.181429 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:31.181434 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:31.181570 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:31.182039 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:31.182046 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:31.182276 (+ 230us) client_negotiation.cc:769] Sending connection context
0926 16:42:31.182461 (+ 185us) client_negotiation.cc:240] Negotiation successful
0926 16:42:31.182758 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":103,"threads_started":1}
I20260926 16:42:31.184831 25828 heartbeater.cc:347] Connected to a master server at 127.21.12.126:43167
I20260926 16:42:31.185089 25828 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.66@KRBTEST.COM" created.
I20260926 16:42:31.185575 25828 heartbeater.cc:511] Master 127.21.12.126:43167 requested a full tablet report, sending...
I20260926 16:42:31.187213 25638 ts_manager.cc:194] Registered new tserver with Master: bd493bcb747c47609b20e8ac0ab6864b (127.21.12.65:41435)
I20260926 16:42:31.188998 25638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.65@KRBTEST.COM'} at 127.21.12.65:50743
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab.
Entry for principal kudu/127.21.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab.
I20260926 16:42:31.200794 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.66:0
--local_ip_for_outbound_sockets=127.21.12.66
--webserver_interface=127.21.12.66
--webserver_port=0
--tserver_master_addrs=127.21.12.126:43167
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab
--principal=kudu/127.21.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:39549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:31.306260 25833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:31.306493 25833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:31.306574 25833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:31.310541 25833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:31.310590 25833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:31.310696 25833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.66
I20260926 16:42:31.315653 25833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:39549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab
--principal=kudu/127.21.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.66
--webserver_port=0
--tserver_master_addrs=127.21.12.126:43167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.66
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:31.316567 25833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:31.317502 25833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:31.326563 25841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:31.326534 25838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:31.326578 25839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:31.326889 25833 server_base.cc:1034] running on GCE node
I20260926 16:42:31.327263 25833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:31.327879 25833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:31.329088 25833 hybrid_clock.cc:648] HybridClock initialized: now 1790440951329072 us; error 25 us; skew 500 ppm
Sep 26 16:42:31 dist-test-slave-n1hw krb5kdc[25594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440951, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:31.332231 25833 init.cc:377] Logged in from keytab as kudu/127.21.12.66@KRBTEST.COM (short username kudu)
I20260926 16:42:31.332902 25833 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:31.333001 25833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:31.333222 25833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:31.335218 25833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-1/data/instance:
uuid: "5099c032f22542398d17c1b62a73053d"
format_stamp: "Formatted at 2026-09-26 16:42:31 on dist-test-slave-n1hw"
server_key: "e844ed7e88b39fc5f7715dd9dca8c16f"
server_key_iv: "83ebe6e81e23f49d2e35d320806138ab"
server_key_version: "encryptionkey@0"
I20260926 16:42:31.335776 25833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance:
uuid: "5099c032f22542398d17c1b62a73053d"
format_stamp: "Formatted at 2026-09-26 16:42:31 on dist-test-slave-n1hw"
server_key: "e844ed7e88b39fc5f7715dd9dca8c16f"
server_key_iv: "83ebe6e81e23f49d2e35d320806138ab"
server_key_version: "encryptionkey@0"
I20260926 16:42:31.339345 25833 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260926 16:42:31.341586 25847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:31.342816 25833 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:42:31.342959 25833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-1/wal
uuid: "5099c032f22542398d17c1b62a73053d"
format_stamp: "Formatted at 2026-09-26 16:42:31 on dist-test-slave-n1hw"
server_key: "e844ed7e88b39fc5f7715dd9dca8c16f"
server_key_iv: "83ebe6e81e23f49d2e35d320806138ab"
server_key_version: "encryptionkey@0"
I20260926 16:42:31.343065 25833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:31.354916 25833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:31.358157 25833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:31.358335 25833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:31.358906 25833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:42:31.359715 25833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:31.359797 25833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:31.359866 25833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:31.359901 25833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:31.368233 25833 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.66:33251
I20260926 16:42:31.368294 25960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.66:33251 every 8 connection(s)
I20260926 16:42:31.368811 25833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
I20260926 16:42:31.376266 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 25833
I20260926 16:42:31.376403 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance
I20260926 16:42:31.376691 21553 external_mini_cluster.cc:1428] Setting key c26ec754a299b5efdd5b77f3f682eb45
Sep 26 16:42:31 dist-test-slave-n1hw krb5kdc[25594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440951, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.66@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:42:31.382341 25677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:31.370492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:58137 (local address 127.21.12.126:43167)
0926 16:42:31.370644 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:31.370649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:31.371431 (+ 782us) server_negotiation.cc:407] Connection header received
0926 16:42:31.372599 (+ 1168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:31.372602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:31.372648 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:31.372721 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:31.373816 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:31.374402 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:31.375479 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:31.375684 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:31.375757 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:31.378368 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:31.378373 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:31.378379 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:31.378425 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:31.379998 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:31.380573 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:31.380575 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:31.380577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:31.380647 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:31.381041 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:31.381046 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:31.381048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:31.381278 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:31.381375 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:31.381965 (+ 590us) server_negotiation.cc:299] Negotiation successful
0926 16:42:31.382112 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 16:42:31.383037 25963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:31.370869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43167 (local address 127.21.12.66:58137)
0926 16:42:31.371279 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:31.371318 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:31.372408 (+ 1090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:31.372874 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:31.372880 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:31.373213 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:31.373652 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:31.373665 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:31.374525 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:31.374528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:31.375373 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:31.375382 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:31.375800 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:31.375803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:31.375950 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:31.376544 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:31.376559 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:31.378209 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:31.380108 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:31.380114 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:31.380129 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:31.380478 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:31.380739 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:31.380744 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:31.380745 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:31.380938 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:31.381388 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:31.381392 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:31.381604 (+ 212us) client_negotiation.cc:769] Sending connection context
0926 16:42:31.381893 (+ 289us) client_negotiation.cc:240] Negotiation successful
0926 16:42:31.382140 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":101,"threads_started":1}
I20260926 16:42:31.384145 25961 heartbeater.cc:347] Connected to a master server at 127.21.12.126:43167
I20260926 16:42:31.384383 25961 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:31.384792 25961 heartbeater.cc:511] Master 127.21.12.126:43167 requested a full tablet report, sending...
I20260926 16:42:31.385833 25638 ts_manager.cc:194] Registered new tserver with Master: 5099c032f22542398d17c1b62a73053d (127.21.12.66:33251)
WARNING: no policy specified for kudu/127.21.12.67@KRBTEST.COM; defaulting to no policy
I20260926 16:42:31.386593 25638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.66@KRBTEST.COM'} at 127.21.12.66:58137
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab.
Entry for principal kudu/127.21.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab.
I20260926 16:42:31.405274 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.67:0
--local_ip_for_outbound_sockets=127.21.12.67
--webserver_interface=127.21.12.67
--webserver_port=0
--tserver_master_addrs=127.21.12.126:43167
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab
--principal=kudu/127.21.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:39549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:31.511514 25966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:31.511736 25966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:31.511816 25966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:31.515774 25966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:31.515820 25966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:31.515928 25966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.67
I20260926 16:42:31.520905 25966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:39549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab
--principal=kudu/127.21.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.67
--webserver_port=0
--tserver_master_addrs=127.21.12.126:43167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.67
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:31.521850 25966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:31.522784 25966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 16:42:31.532222 25966 server_base.cc:1034] running on GCE node
W20260926 16:42:31.532191 25971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:31.532178 25972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:31.532178 25974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:31.532819 25966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:31.533437 25966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:31.534586 25966 hybrid_clock.cc:648] HybridClock initialized: now 1790440951534542 us; error 54 us; skew 500 ppm
Sep 26 16:42:31 dist-test-slave-n1hw krb5kdc[25594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440951, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:31.537189 25966 init.cc:377] Logged in from keytab as kudu/127.21.12.67@KRBTEST.COM (short username kudu)
I20260926 16:42:31.537899 25966 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:31.537971 25966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:31.538130 25966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:31.539986 25966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-2/data/instance:
uuid: "c0e9a2271537482fa3d6d4ae5e95eb13"
format_stamp: "Formatted at 2026-09-26 16:42:31 on dist-test-slave-n1hw"
server_key: "abd6a87e5e8ff64e22d156fb89e12196"
server_key_iv: "f8c8e0c3da9f28a674a199a40d9f5eab"
server_key_version: "encryptionkey@0"
I20260926 16:42:31.540472 25966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance:
uuid: "c0e9a2271537482fa3d6d4ae5e95eb13"
format_stamp: "Formatted at 2026-09-26 16:42:31 on dist-test-slave-n1hw"
server_key: "abd6a87e5e8ff64e22d156fb89e12196"
server_key_iv: "f8c8e0c3da9f28a674a199a40d9f5eab"
server_key_version: "encryptionkey@0"
I20260926 16:42:31.544061 25966 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 16:42:31.546402 25980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:31.547613 25966 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:42:31.547739 25966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-2/wal
uuid: "c0e9a2271537482fa3d6d4ae5e95eb13"
format_stamp: "Formatted at 2026-09-26 16:42:31 on dist-test-slave-n1hw"
server_key: "abd6a87e5e8ff64e22d156fb89e12196"
server_key_iv: "f8c8e0c3da9f28a674a199a40d9f5eab"
server_key_version: "encryptionkey@0"
I20260926 16:42:31.547834 25966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:31.586122 25966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:31.589093 25966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:31.589295 25966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:31.589895 25966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:42:31.590939 25966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:31.590988 25966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:31.591054 25966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:31.591087 25966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:31.599937 25966 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.67:36561
I20260926 16:42:31.599959 26093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.67:36561 every 8 connection(s)
I20260926 16:42:31.600514 25966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
I20260926 16:42:31.600983 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 25966
I20260926 16:42:31.601117 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance
I20260926 16:42:31.601362 21553 external_mini_cluster.cc:1428] Setting key 81fc825474a5dc6408fb7cd1a3cb0bbc
Sep 26 16:42:31 dist-test-slave-n1hw krb5kdc[25594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440951, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.67@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:42:31.613881 25677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:31.602138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:33071 (local address 127.21.12.126:43167)
0926 16:42:31.602240 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:31.602245 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:31.603734 (+ 1489us) server_negotiation.cc:407] Connection header received
0926 16:42:31.604997 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:31.605000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:31.605054 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:31.605130 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:31.606322 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:31.606852 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:31.607988 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:31.608212 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:31.608323 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:31.610615 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:31.610618 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:31.610621 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:31.610647 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:31.611864 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:31.612351 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:31.612356 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:31.612358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:31.612460 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:31.612798 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:31.612803 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:31.612805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:31.612987 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:31.613093 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:31.613559 (+ 466us) server_negotiation.cc:299] Negotiation successful
0926 16:42:31.613697 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 16:42:31.614776 26096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:31.602720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43167 (local address 127.21.12.67:33071)
0926 16:42:31.603596 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:31.603650 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:31.604781 (+ 1131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:31.605312 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:31.605318 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:31.605651 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:31.606184 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:31.606197 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:31.606982 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:31.606986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:31.607842 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:31.607851 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:31.608354 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:31.608357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:31.608525 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:31.609027 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:31.609042 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:31.610457 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:31.611983 (+ 1526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:31.611988 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:31.612001 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:31.612246 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:31.612561 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:31.612566 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:31.612568 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:31.612701 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:31.613101 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:31.613109 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:31.613330 (+ 221us) client_negotiation.cc:769] Sending connection context
0926 16:42:31.613506 (+ 176us) client_negotiation.cc:240] Negotiation successful
0926 16:42:31.613822 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":98,"threads_started":1}
I20260926 16:42:31.615896 26094 heartbeater.cc:347] Connected to a master server at 127.21.12.126:43167
I20260926 16:42:31.616132 26094 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:31.616569 26094 heartbeater.cc:511] Master 127.21.12.126:43167 requested a full tablet report, sending...
I20260926 16:42:31.617513 25638 ts_manager.cc:194] Registered new tserver with Master: c0e9a2271537482fa3d6d4ae5e95eb13 (127.21.12.67:36561)
I20260926 16:42:31.618104 25638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.67@KRBTEST.COM'} at 127.21.12.67:33071
I20260926 16:42:31.627218 21553 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:42:31.636030 25677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:31.629163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58814 (local address 127.21.12.126:43167)
0926 16:42:31.629312 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:31.629317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:31.629476 (+ 159us) server_negotiation.cc:407] Connection header received
0926 16:42:31.629563 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:31.629565 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:31.629603 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:31.629672 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:31.630362 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:31.630843 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:31.631626 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:31.631837 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:31.631935 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:31.633299 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:31.633302 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:31.633304 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:31.633326 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:31.634594 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:31.634982 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:31.634985 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:31.634986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:31.635054 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:31.635271 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:31.635273 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:31.635274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:31.635463 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:31.635560 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:31.635741 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 16:42:31.635877 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 16:42:31.639025 25638 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58814:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:42:31.641988 25638 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:42:31.656958 26106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:31.650516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58990 (local address 127.21.12.65:41435)
0926 16:42:31.650942 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:31.650949 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:31.651331 (+ 382us) server_negotiation.cc:407] Connection header received
0926 16:42:31.651420 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:31.651427 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:31.651666 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:31.651830 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:31.652437 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:31.653659 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:31.655357 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:31.656123 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:31.656233 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:31.656386 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:31.656674 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 16:42:31.656782 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":100,"threads_started":1}
I20260926 16:42:31.656955 26105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:31.650485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.67:36561 (local address 127.0.0.1:43868)
0926 16:42:31.651077 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:31.651094 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:31.651180 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:31.652644 (+ 1464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:31.652647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:31.652668 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:31.652771 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:31.652777 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:31.654097 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:31.654101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:31.655313 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:31.655323 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:31.656329 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:31.656332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:31.656401 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:31.656498 (+ 97us) client_negotiation.cc:769] Sending connection context
0926 16:42:31.656609 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 16:42:31.656787 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"mutex_wait_us":81,"thread_start_us":140,"threads_started":1}
I20260926 16:42:31.657320 26108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:31.651193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:33251 (local address 127.0.0.1:52250)
0926 16:42:31.651920 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:31.651935 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:31.652021 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:31.653035 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:31.653038 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:31.653055 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:31.653148 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:31.653154 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:31.654318 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:31.654321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:31.655708 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:31.655717 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:31.656915 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:31.656918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:31.657021 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:31.657048 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 16:42:31.657130 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 16:42:31.657201 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"mutex_wait_us":2,"thread_start_us":97,"threads_started":1}
I20260926 16:42:31.657336 26109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:31.651285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43868 (local address 127.21.12.67:36561)
0926 16:42:31.651799 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:31.651811 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:31.651832 (+ 21us) server_negotiation.cc:407] Connection header received
0926 16:42:31.651906 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:31.651911 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:31.652490 (+ 579us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:31.652717 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:31.652872 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:31.653960 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:31.655470 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:31.656175 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:31.656854 (+ 679us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:31.657013 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:31.657070 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 16:42:31.657173 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":177,"threads_started":1}
I20260926 16:42:31.657853 26110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:31.651531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52250 (local address 127.21.12.66:33251)
0926 16:42:31.652557 (+ 1026us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:31.652567 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:31.652613 (+ 46us) server_negotiation.cc:407] Connection header received
0926 16:42:31.652688 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:31.652692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:31.652889 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:31.653060 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:31.653243 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:31.654179 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:31.655838 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:31.656607 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:31.656710 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:31.656838 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:31.657566 (+ 728us) server_negotiation.cc:299] Negotiation successful
0926 16:42:31.657665 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":856,"thread_start_us":117,"threads_started":1}
I20260926 16:42:31.660315 26028 tablet_service.cc:1467] Processing CreateTablet for tablet ec4e037bb9a84e118a12501e29e682c1 (DEFAULT_TABLE table=test-table [id=566bd7cb20e24c23a07309ce5e77e1e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:31.660315 25895 tablet_service.cc:1467] Processing CreateTablet for tablet ec4e037bb9a84e118a12501e29e682c1 (DEFAULT_TABLE table=test-table [id=566bd7cb20e24c23a07309ce5e77e1e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:31.661254 25895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec4e037bb9a84e118a12501e29e682c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:31.661264 26028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec4e037bb9a84e118a12501e29e682c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:31.662443 26107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:31.650413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:41435 (local address 127.0.0.1:58990)
0926 16:42:31.650902 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:31.650968 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:31.651118 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:31.652041 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:31.652047 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:31.652068 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:31.652212 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:31.652219 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:31.653812 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:31.653816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:31.655180 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:31.655213 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:31.656267 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:31.656272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:31.656398 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:31.656498 (+ 100us) client_negotiation.cc:769] Sending connection context
0926 16:42:31.656616 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 16:42:31.656787 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":82,"thread_start_us":80,"threads_started":1}
I20260926 16:42:31.665333 25762 tablet_service.cc:1467] Processing CreateTablet for tablet ec4e037bb9a84e118a12501e29e682c1 (DEFAULT_TABLE table=test-table [id=566bd7cb20e24c23a07309ce5e77e1e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:31.666350 25762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec4e037bb9a84e118a12501e29e682c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:31.667954 26111 tablet_bootstrap.cc:492] T ec4e037bb9a84e118a12501e29e682c1 P c0e9a2271537482fa3d6d4ae5e95eb13: Bootstrap starting.
I20260926 16:42:31.668344 26112 tablet_bootstrap.cc:492] T ec4e037bb9a84e118a12501e29e682c1 P 5099c032f22542398d17c1b62a73053d: Bootstrap starting.
I20260926 16:42:31.670305 26112 tablet_bootstrap.cc:654] T ec4e037bb9a84e118a12501e29e682c1 P 5099c032f22542398d17c1b62a73053d: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:31.671581 26112 log.cc:824] T ec4e037bb9a84e118a12501e29e682c1 P 5099c032f22542398d17c1b62a73053d: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:31.671916 26111 tablet_bootstrap.cc:654] T ec4e037bb9a84e118a12501e29e682c1 P c0e9a2271537482fa3d6d4ae5e95eb13: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:31.672910 26114 tablet_bootstrap.cc:492] T ec4e037bb9a84e118a12501e29e682c1 P bd493bcb747c47609b20e8ac0ab6864b: Bootstrap starting.
I20260926 16:42:31.673447 26111 log.cc:824] T ec4e037bb9a84e118a12501e29e682c1 P c0e9a2271537482fa3d6d4ae5e95eb13: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:31.673738 26112 tablet_bootstrap.cc:492] T ec4e037bb9a84e118a12501e29e682c1 P 5099c032f22542398d17c1b62a73053d: No bootstrap required, opened a new log
I20260926 16:42:31.673954 26112 ts_tablet_manager.cc:1397] T ec4e037bb9a84e118a12501e29e682c1 P 5099c032f22542398d17c1b62a73053d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.002s
I20260926 16:42:31.674787 26114 tablet_bootstrap.cc:654] T ec4e037bb9a84e118a12501e29e682c1 P bd493bcb747c47609b20e8ac0ab6864b: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:31.675329 26111 tablet_bootstrap.cc:492] T ec4e037bb9a84e118a12501e29e682c1 P c0e9a2271537482fa3d6d4ae5e95eb13: No bootstrap required, opened a new log
I20260926 16:42:31.675504 26111 ts_tablet_manager.cc:1397] T ec4e037bb9a84e118a12501e29e682c1 P c0e9a2271537482fa3d6d4ae5e95eb13: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.001s
I20260926 16:42:31.675823 26114 log.cc:824] T ec4e037bb9a84e118a12501e29e682c1 P bd493bcb747c47609b20e8ac0ab6864b: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:31.677244 26114 tablet_bootstrap.cc:492] T ec4e037bb9a84e118a12501e29e682c1 P bd493bcb747c47609b20e8ac0ab6864b: No bootstrap required, opened a new log
I20260926 16:42:31.677428 26114 ts_tablet_manager.cc:1397] T ec4e037bb9a84e118a12501e29e682c1 P bd493bcb747c47609b20e8ac0ab6864b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 16:42:31.677824 26112 raft_consensus.cc:348] T ec4e037bb9a84e118a12501e29e682c1 P 5099c032f22542398d17c1b62a73053d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd493bcb747c47609b20e8ac0ab6864b" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41435 } } peers { permanent_uuid: "c0e9a2271537482fa3d6d4ae5e95eb13" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36561 } } peers { permanent_uuid: "5099c032f22542398d17c1b62a73053d" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 33251 } }
I20260926 16:42:31.678103 26112 raft_consensus.cc:374] T ec4e037bb9a84e118a12501e29e682c1 P 5099c032f22542398d17c1b62a73053d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:31.678196 26112 raft_consensus.cc:729] T ec4e037bb9a84e118a12501e29e682c1 P 5099c032f22542398d17c1b62a73053d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5099c032f22542398d17c1b62a73053d, State: Initialized, Role: FOLLOWER
I20260926 16:42:31.678653 26112 consensus_queue.cc:260] T ec4e037bb9a84e118a12501e29e682c1 P 5099c032f22542398d17c1b62a73053d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd493bcb747c47609b20e8ac0ab6864b" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41435 } } peers { permanent_uuid: "c0e9a2271537482fa3d6d4ae5e95eb13" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36561 } } peers { permanent_uuid: "5099c032f22542398d17c1b62a73053d" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 33251 } }
I20260926 16:42:31.679324 25961 heartbeater.cc:503] Master 127.21.12.126:43167 was elected leader, sending a full tablet report...
I20260926 16:42:31.679605 26112 ts_tablet_manager.cc:1428] T ec4e037bb9a84e118a12501e29e682c1 P 5099c032f22542398d17c1b62a73053d: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260926 16:42:31.680379 26111 raft_consensus.cc:348] T ec4e037bb9a84e118a12501e29e682c1 P c0e9a2271537482fa3d6d4ae5e95eb13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd493bcb747c47609b20e8ac0ab6864b" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41435 } } peers { permanent_uuid: "c0e9a2271537482fa3d6d4ae5e95eb13" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36561 } } peers { permanent_uuid: "5099c032f22542398d17c1b62a73053d" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 33251 } }
I20260926 16:42:31.680670 26111 raft_consensus.cc:374] T ec4e037bb9a84e118a12501e29e682c1 P c0e9a2271537482fa3d6d4ae5e95eb13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:31.680737 26111 raft_consensus.cc:729] T ec4e037bb9a84e118a12501e29e682c1 P c0e9a2271537482fa3d6d4ae5e95eb13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0e9a2271537482fa3d6d4ae5e95eb13, State: Initialized, Role: FOLLOWER
I20260926 16:42:31.680953 26114 raft_consensus.cc:348] T ec4e037bb9a84e118a12501e29e682c1 P bd493bcb747c47609b20e8ac0ab6864b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd493bcb747c47609b20e8ac0ab6864b" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41435 } } peers { permanent_uuid: "c0e9a2271537482fa3d6d4ae5e95eb13" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36561 } } peers { permanent_uuid: "5099c032f22542398d17c1b62a73053d" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 33251 } }
I20260926 16:42:31.681185 26114 raft_consensus.cc:374] T ec4e037bb9a84e118a12501e29e682c1 P bd493bcb747c47609b20e8ac0ab6864b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:31.681152 26111 consensus_queue.cc:260] T ec4e037bb9a84e118a12501e29e682c1 P c0e9a2271537482fa3d6d4ae5e95eb13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd493bcb747c47609b20e8ac0ab6864b" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41435 } } peers { permanent_uuid: "c0e9a2271537482fa3d6d4ae5e95eb13" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36561 } } peers { permanent_uuid: "5099c032f22542398d17c1b62a73053d" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 33251 } }
I20260926 16:42:31.681274 26114 raft_consensus.cc:729] T ec4e037bb9a84e118a12501e29e682c1 P bd493bcb747c47609b20e8ac0ab6864b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd493bcb747c47609b20e8ac0ab6864b, State: Initialized, Role: FOLLOWER
I20260926 16:42:31.681696 26114 consensus_queue.cc:260] T ec4e037bb9a84e118a12501e29e682c1 P bd493bcb747c47609b20e8ac0ab6864b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd493bcb747c47609b20e8ac0ab6864b" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41435 } } peers { permanent_uuid: "c0e9a2271537482fa3d6d4ae5e95eb13" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36561 } } peers { permanent_uuid: "5099c032f22542398d17c1b62a73053d" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 33251 } }
I20260926 16:42:31.681793 26094 heartbeater.cc:503] Master 127.21.12.126:43167 was elected leader, sending a full tablet report...
I20260926 16:42:31.682163 26111 ts_tablet_manager.cc:1428] T ec4e037bb9a84e118a12501e29e682c1 P c0e9a2271537482fa3d6d4ae5e95eb13: Time spent starting tablet: real 0.007s user 0.002s sys 0.005s
I20260926 16:42:31.682569 26114 ts_tablet_manager.cc:1428] T ec4e037bb9a84e118a12501e29e682c1 P bd493bcb747c47609b20e8ac0ab6864b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 16:42:31.683264 25828 heartbeater.cc:503] Master 127.21.12.126:43167 was elected leader, sending a full tablet report...
I20260926 16:42:31.787178 26117 raft_consensus.cc:482] T ec4e037bb9a84e118a12501e29e682c1 P 5099c032f22542398d17c1b62a73053d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:42:31.787367 26117 raft_consensus.cc:504] T ec4e037bb9a84e118a12501e29e682c1 P 5099c032f22542398d17c1b62a73053d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd493bcb747c47609b20e8ac0ab6864b" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41435 } } peers { permanent_uuid: "c0e9a2271537482fa3d6d4ae5e95eb13" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36561 } } peers { permanent_uuid: "5099c032f22542398d17c1b62a73053d" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 33251 } }
I20260926 16:42:31.788455 26117 leader_election.cc:288] T ec4e037bb9a84e118a12501e29e682c1 P 5099c032f22542398d17c1b62a73053d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd493bcb747c47609b20e8ac0ab6864b (127.21.12.65:41435), c0e9a2271537482fa3d6d4ae5e95eb13 (127.21.12.67:36561)
I20260926 16:42:31.792707 25963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:31.788719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.67:36561 (local address 127.21.12.66:37531)
0926 16:42:31.788881 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:31.788899 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:31.789066 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:31.789373 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:31.789376 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:31.789394 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:31.789514 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:31.789519 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:31.790247 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:31.790253 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:31.791224 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:31.791253 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:31.792283 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:31.792287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:31.792395 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:31.792419 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 16:42:31.792499 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 16:42:31.792569 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 16:42:31.792759 26109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:31.788861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:37531 (local address 127.21.12.67:36561)
0926 16:42:31.789022 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:31.789028 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:31.789047 (+ 19us) server_negotiation.cc:407] Connection header received
0926 16:42:31.789201 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:31.789205 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:31.789259 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:31.789352 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:31.789623 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:31.790139 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:31.791389 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:31.792059 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:31.792174 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:31.792256 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:31.792538 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 16:42:31.792631 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 16:42:31.793591 26120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:31.788719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:41435 (local address 127.21.12.66:53331)
0926 16:42:31.789413 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:31.789426 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:31.789559 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:31.789862 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:31.789865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:31.789881 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:31.790003 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:31.790008 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:31.790875 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:31.790878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:31.792038 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:31.792048 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:31.793189 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:31.793193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:31.793290 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:31.793314 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 16:42:31.793392 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 16:42:31.793451 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"mutex_wait_us":301,"thread_start_us":148,"threads_started":1}
I20260926 16:42:31.793725 26106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:31.788935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:53331 (local address 127.21.12.65:41435)
0926 16:42:31.789114 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:31.789121 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:31.789527 (+ 406us) server_negotiation.cc:407] Connection header received
0926 16:42:31.789680 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:31.789683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:31.789732 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:31.789845 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:31.790117 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:31.790755 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:31.792190 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:31.792987 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:31.793092 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:31.793147 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:31.793414 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 16:42:31.793538 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 16:42:31.793602 26048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec4e037bb9a84e118a12501e29e682c1" candidate_uuid: "5099c032f22542398d17c1b62a73053d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0e9a2271537482fa3d6d4ae5e95eb13" is_pre_election: true
I20260926 16:42:31.794044 26048 raft_consensus.cc:2454] T ec4e037bb9a84e118a12501e29e682c1 P c0e9a2271537482fa3d6d4ae5e95eb13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5099c032f22542398d17c1b62a73053d in term 0.
I20260926 16:42:31.794531 25782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec4e037bb9a84e118a12501e29e682c1" candidate_uuid: "5099c032f22542398d17c1b62a73053d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd493bcb747c47609b20e8ac0ab6864b" is_pre_election: true
I20260926 16:42:31.794577 25851 leader_election.cc:302] T ec4e037bb9a84e118a12501e29e682c1 P 5099c032f22542398d17c1b62a73053d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5099c032f22542398d17c1b62a73053d, c0e9a2271537482fa3d6d4ae5e95eb13; no voters:
I20260926 16:42:31.794823 25782 raft_consensus.cc:2454] T ec4e037bb9a84e118a12501e29e682c1 P bd493bcb747c47609b20e8ac0ab6864b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5099c032f22542398d17c1b62a73053d in term 0.
I20260926 16:42:31.794832 26117 raft_consensus.cc:2781] T ec4e037bb9a84e118a12501e29e682c1 P 5099c032f22542398d17c1b62a73053d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:42:31.794919 26117 raft_consensus.cc:482] T ec4e037bb9a84e118a12501e29e682c1 P 5099c032f22542398d17c1b62a73053d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:42:31.794977 26117 raft_consensus.cc:3037] T ec4e037bb9a84e118a12501e29e682c1 P 5099c032f22542398d17c1b62a73053d [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:31.796001 26117 raft_consensus.cc:504] T ec4e037bb9a84e118a12501e29e682c1 P 5099c032f22542398d17c1b62a73053d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd493bcb747c47609b20e8ac0ab6864b" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41435 } } peers { permanent_uuid: "c0e9a2271537482fa3d6d4ae5e95eb13" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36561 } } peers { permanent_uuid: "5099c032f22542398d17c1b62a73053d" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 33251 } }
I20260926 16:42:31.796362 26117 leader_election.cc:288] T ec4e037bb9a84e118a12501e29e682c1 P 5099c032f22542398d17c1b62a73053d [CANDIDATE]: Term 1 election: Requested vote from peers bd493bcb747c47609b20e8ac0ab6864b (127.21.12.65:41435), c0e9a2271537482fa3d6d4ae5e95eb13 (127.21.12.67:36561)
I20260926 16:42:31.796656 26048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec4e037bb9a84e118a12501e29e682c1" candidate_uuid: "5099c032f22542398d17c1b62a73053d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0e9a2271537482fa3d6d4ae5e95eb13"
I20260926 16:42:31.796716 25782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec4e037bb9a84e118a12501e29e682c1" candidate_uuid: "5099c032f22542398d17c1b62a73053d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd493bcb747c47609b20e8ac0ab6864b"
I20260926 16:42:31.796808 26048 raft_consensus.cc:3037] T ec4e037bb9a84e118a12501e29e682c1 P c0e9a2271537482fa3d6d4ae5e95eb13 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:31.796824 25782 raft_consensus.cc:3037] T ec4e037bb9a84e118a12501e29e682c1 P bd493bcb747c47609b20e8ac0ab6864b [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:31.797749 25782 raft_consensus.cc:2454] T ec4e037bb9a84e118a12501e29e682c1 P bd493bcb747c47609b20e8ac0ab6864b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5099c032f22542398d17c1b62a73053d in term 1.
I20260926 16:42:31.798051 25849 leader_election.cc:302] T ec4e037bb9a84e118a12501e29e682c1 P 5099c032f22542398d17c1b62a73053d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5099c032f22542398d17c1b62a73053d, bd493bcb747c47609b20e8ac0ab6864b; no voters:
I20260926 16:42:31.798090 26048 raft_consensus.cc:2454] T ec4e037bb9a84e118a12501e29e682c1 P c0e9a2271537482fa3d6d4ae5e95eb13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5099c032f22542398d17c1b62a73053d in term 1.
I20260926 16:42:31.798218 26117 raft_consensus.cc:2781] T ec4e037bb9a84e118a12501e29e682c1 P 5099c032f22542398d17c1b62a73053d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:31.798463 26117 raft_consensus.cc:686] T ec4e037bb9a84e118a12501e29e682c1 P 5099c032f22542398d17c1b62a73053d [term 1 LEADER]: Becoming Leader. State: Replica: 5099c032f22542398d17c1b62a73053d, State: Running, Role: LEADER
I20260926 16:42:31.798764 26117 consensus_queue.cc:237] T ec4e037bb9a84e118a12501e29e682c1 P 5099c032f22542398d17c1b62a73053d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd493bcb747c47609b20e8ac0ab6864b" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41435 } } peers { permanent_uuid: "c0e9a2271537482fa3d6d4ae5e95eb13" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36561 } } peers { permanent_uuid: "5099c032f22542398d17c1b62a73053d" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 33251 } }
I20260926 16:42:31.801793 25638 catalog_manager.cc:5495] T ec4e037bb9a84e118a12501e29e682c1 P 5099c032f22542398d17c1b62a73053d reported cstate change: term changed from 0 to 1, leader changed from <none> to 5099c032f22542398d17c1b62a73053d (127.21.12.66). New cstate: current_term: 1 leader_uuid: "5099c032f22542398d17c1b62a73053d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd493bcb747c47609b20e8ac0ab6864b" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0e9a2271537482fa3d6d4ae5e95eb13" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5099c032f22542398d17c1b62a73053d" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 33251 } health_report { overall_health: HEALTHY } } }
I20260926 16:42:31.844409 26110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:31.840910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52258 (local address 127.21.12.66:33251)
0926 16:42:31.841044 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:31.841048 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:31.841062 (+ 14us) server_negotiation.cc:407] Connection header received
0926 16:42:31.841243 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:31.841246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:31.841299 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:31.841404 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:42:31.841682 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:31.842258 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:31.843070 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:31.843272 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:31.843348 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:31.843627 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:42:31.844083 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:42:31.844171 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:31.844248 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 16:42:31.844307 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 16:42:31.850255 25782 raft_consensus.cc:1261] T ec4e037bb9a84e118a12501e29e682c1 P bd493bcb747c47609b20e8ac0ab6864b [term 1 FOLLOWER]: Refusing update from remote peer 5099c032f22542398d17c1b62a73053d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:42:31.850255 26048 raft_consensus.cc:1261] T ec4e037bb9a84e118a12501e29e682c1 P c0e9a2271537482fa3d6d4ae5e95eb13 [term 1 FOLLOWER]: Refusing update from remote peer 5099c032f22542398d17c1b62a73053d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:42:31.850981 26121 consensus_queue.cc:1035] T ec4e037bb9a84e118a12501e29e682c1 P 5099c032f22542398d17c1b62a73053d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd493bcb747c47609b20e8ac0ab6864b" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:42:31.851186 26117 consensus_queue.cc:1035] T ec4e037bb9a84e118a12501e29e682c1 P 5099c032f22542398d17c1b62a73053d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0e9a2271537482fa3d6d4ae5e95eb13" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 16:42:31.851364 26095 tablet.cc:2369] T ec4e037bb9a84e118a12501e29e682c1 P c0e9a2271537482fa3d6d4ae5e95eb13: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:42:31.858016 26124 mvcc.cc:205] Tried to move back new op lower bound from 7333646138771496960 to 7333646138570854400. Current Snapshot: MvccSnapshot[applied={T|T < 7333646138771496960}]
I20260926 16:42:31.858572 26125 mvcc.cc:205] Tried to move back new op lower bound from 7333646138771496960 to 7333646138570854400. Current Snapshot: MvccSnapshot[applied={T|T < 7333646138771496960}]
I20260926 16:42:31.859556 26126 mvcc.cc:205] Tried to move back new op lower bound from 7333646138771496960 to 7333646138570854400. Current Snapshot: MvccSnapshot[applied={T|T < 7333646138771496960}]
I20260926 16:42:31.868541 25638 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:42:31.868705 25638 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:42:31.871214 25638 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 493fe9e446a846958b679d0832e6b476: Sending DeleteTablet for 3 replicas of tablet ec4e037bb9a84e118a12501e29e682c1
I20260926 16:42:31.871966 25895 tablet_service.cc:1514] Processing DeleteTablet for tablet ec4e037bb9a84e118a12501e29e682c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:31 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:52250
I20260926 16:42:31.871964 26028 tablet_service.cc:1514] Processing DeleteTablet for tablet ec4e037bb9a84e118a12501e29e682c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:31 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:43868
I20260926 16:42:31.872047 25762 tablet_service.cc:1514] Processing DeleteTablet for tablet ec4e037bb9a84e118a12501e29e682c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:31 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:58990
I20260926 16:42:31.872418 26135 tablet_replica.cc:330] T ec4e037bb9a84e118a12501e29e682c1 P 5099c032f22542398d17c1b62a73053d: stopping tablet replica
I20260926 16:42:31.872427 26134 tablet_replica.cc:330] T ec4e037bb9a84e118a12501e29e682c1 P c0e9a2271537482fa3d6d4ae5e95eb13: stopping tablet replica
I20260926 16:42:31.872701 26135 raft_consensus.cc:2229] T ec4e037bb9a84e118a12501e29e682c1 P 5099c032f22542398d17c1b62a73053d [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:42:31.872867 26136 tablet_replica.cc:330] T ec4e037bb9a84e118a12501e29e682c1 P bd493bcb747c47609b20e8ac0ab6864b: stopping tablet replica
I20260926 16:42:31.873061 26135 raft_consensus.cc:2258] T ec4e037bb9a84e118a12501e29e682c1 P 5099c032f22542398d17c1b62a73053d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:42:31.873131 26136 raft_consensus.cc:2229] T ec4e037bb9a84e118a12501e29e682c1 P bd493bcb747c47609b20e8ac0ab6864b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:42:31.873360 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 25700
I20260926 16:42:31.873404 26136 raft_consensus.cc:2258] T ec4e037bb9a84e118a12501e29e682c1 P bd493bcb747c47609b20e8ac0ab6864b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:42:31.875840 26135 ts_tablet_manager.cc:1905] T ec4e037bb9a84e118a12501e29e682c1 P 5099c032f22542398d17c1b62a73053d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:42:31.879277 26135 ts_tablet_manager.cc:1918] T ec4e037bb9a84e118a12501e29e682c1 P 5099c032f22542398d17c1b62a73053d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:42:31.879364 26135 log.cc:1196] T ec4e037bb9a84e118a12501e29e682c1 P 5099c032f22542398d17c1b62a73053d: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440924937351-21553-0/minicluster-data/ts-1/wal/wals/ec4e037bb9a84e118a12501e29e682c1
I20260926 16:42:31.879678 26135 ts_tablet_manager.cc:1939] T ec4e037bb9a84e118a12501e29e682c1 P 5099c032f22542398d17c1b62a73053d: Deleting consensus metadata
I20260926 16:42:31.880982 25623 catalog_manager.cc:4841] TS 5099c032f22542398d17c1b62a73053d (127.21.12.66:33251): tablet ec4e037bb9a84e118a12501e29e682c1 (table test-table [id=566bd7cb20e24c23a07309ce5e77e1e0]) successfully deleted
W20260926 16:42:31.881970 25623 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.12.65:41435 (error 108)
I20260926 16:42:31.882942 26107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:31.882650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:41435 (local address 127.0.0.1:59004)
0926 16:42:31.882774 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:31.882847 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.65:41435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 16:42:31.883033 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 25833
W20260926 16:42:31.883822 25623 catalog_manager.cc:4584] TS bd493bcb747c47609b20e8ac0ab6864b (127.21.12.65:41435): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ec4e037bb9a84e118a12501e29e682c1: Network error: Client connection negotiation failed: client connection to 127.21.12.65:41435: connect: Connection refused (error 111)
I20260926 16:42:31.872701 26134 raft_consensus.cc:2229] T ec4e037bb9a84e118a12501e29e682c1 P c0e9a2271537482fa3d6d4ae5e95eb13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:42:31.884914 26134 raft_consensus.cc:2258] T ec4e037bb9a84e118a12501e29e682c1 P c0e9a2271537482fa3d6d4ae5e95eb13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:42:31.886322 26134 ts_tablet_manager.cc:1905] T ec4e037bb9a84e118a12501e29e682c1 P c0e9a2271537482fa3d6d4ae5e95eb13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:42:31.888422 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 25966
I20260926 16:42:31.895887 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 25607
2026-09-26T16:42:31Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2336 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T16:42:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:42:31Z Disabled control of system clock
I20260926 16:42:31.925638 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:38755
--webserver_interface=127.21.12.126
--webserver_port=0
--builtin_ntp_servers=127.21.12.84:42541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:38755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 16:42:32.032011 26141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:32.032292 26141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:32.032393 26141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:32.036755 26141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:42:32.036839 26141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:32.036876 26141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:32.036914 26141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:42:32.036949 26141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:42:32.041636 26141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:42541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:38755
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:38755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:32.042641 26141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:32.043985 26141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:32.052224 26147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:32.052248 26146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:32.052275 26149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:32.052402 26141 server_base.cc:1034] running on GCE node
I20260926 16:42:32.052955 26141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:32.054064 26141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:32.055254 26141 hybrid_clock.cc:648] HybridClock initialized: now 1790440952055240 us; error 25 us; skew 500 ppm
I20260926 16:42:32.057120 26141 webserver.cc:460] Webserver started at http://127.21.12.126:45965/ using document root <none> and password file <none>
I20260926 16:42:32.057682 26141 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:32.057793 26141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:32.058027 26141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:32.059976 26141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "65c274fde730484da46cd219df51359d"
format_stamp: "Formatted at 2026-09-26 16:42:32 on dist-test-slave-n1hw"
server_key: "c9761db5d9cf1687a0b21a4507fe95c5"
server_key_iv: "3c9ab22ef253b1dcb14655b54fb86ba1"
server_key_version: "encryptionkey@0"
I20260926 16:42:32.060585 26141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "65c274fde730484da46cd219df51359d"
format_stamp: "Formatted at 2026-09-26 16:42:32 on dist-test-slave-n1hw"
server_key: "c9761db5d9cf1687a0b21a4507fe95c5"
server_key_iv: "3c9ab22ef253b1dcb14655b54fb86ba1"
server_key_version: "encryptionkey@0"
I20260926 16:42:32.064069 26141 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260926 16:42:32.066331 26155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:32.067525 26141 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:42:32.067657 26141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "65c274fde730484da46cd219df51359d"
format_stamp: "Formatted at 2026-09-26 16:42:32 on dist-test-slave-n1hw"
server_key: "c9761db5d9cf1687a0b21a4507fe95c5"
server_key_iv: "3c9ab22ef253b1dcb14655b54fb86ba1"
server_key_version: "encryptionkey@0"
I20260926 16:42:32.067762 26141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:32.072314 26141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:32.072875 26141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:32.073055 26141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:32.079754 26141 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.126:38755
I20260926 16:42:32.079761 26207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.126:38755 every 8 connection(s)
I20260926 16:42:32.080655 26141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:42:32.083612 26208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:32.088915 26208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65c274fde730484da46cd219df51359d: Bootstrap starting.
I20260926 16:42:32.090693 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 26141
I20260926 16:42:32.090833 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:42:32.091115 21553 external_mini_cluster.cc:1428] Setting key e35c379ff3e53cad8a98306f2dd4bfef
I20260926 16:42:32.091441 26208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65c274fde730484da46cd219df51359d: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:32.092306 26208 log.cc:824] T 00000000000000000000000000000000 P 65c274fde730484da46cd219df51359d: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:32.094276 26208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65c274fde730484da46cd219df51359d: No bootstrap required, opened a new log
I20260926 16:42:32.096196 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:32.092190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37504 (local address 127.21.12.126:38755)
0926 16:42:32.092866 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:32.092885 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:32.092927 (+ 42us) server_negotiation.cc:407] Connection header received
0926 16:42:32.093562 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:32.093577 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:32.093911 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:32.094173 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:32.094987 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:32.094992 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:32.095004 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:32.095015 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:32.095067 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:32.095082 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:32.095168 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:32.095296 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 16:42:32.095572 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":113,"threads_started":1}
I20260926 16:42:32.098210 26208 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65c274fde730484da46cd219df51359d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c274fde730484da46cd219df51359d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 38755 } }
I20260926 16:42:32.098519 26208 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65c274fde730484da46cd219df51359d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:32.098582 26208 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65c274fde730484da46cd219df51359d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c274fde730484da46cd219df51359d, State: Initialized, Role: FOLLOWER
I20260926 16:42:32.098964 26208 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65c274fde730484da46cd219df51359d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c274fde730484da46cd219df51359d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 38755 } }
I20260926 16:42:32.099103 26208 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65c274fde730484da46cd219df51359d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:42:32.099164 26208 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65c274fde730484da46cd219df51359d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:42:32.099272 26208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65c274fde730484da46cd219df51359d [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:32.100085 26208 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65c274fde730484da46cd219df51359d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c274fde730484da46cd219df51359d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 38755 } }
I20260926 16:42:32.100411 26208 leader_election.cc:302] T 00000000000000000000000000000000 P 65c274fde730484da46cd219df51359d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65c274fde730484da46cd219df51359d; no voters:
I20260926 16:42:32.100694 26208 leader_election.cc:288] T 00000000000000000000000000000000 P 65c274fde730484da46cd219df51359d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:42:32.100804 26213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65c274fde730484da46cd219df51359d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:32.101039 26213 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65c274fde730484da46cd219df51359d [term 1 LEADER]: Becoming Leader. State: Replica: 65c274fde730484da46cd219df51359d, State: Running, Role: LEADER
I20260926 16:42:32.101320 26213 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65c274fde730484da46cd219df51359d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c274fde730484da46cd219df51359d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 38755 } }
I20260926 16:42:32.101925 26208 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65c274fde730484da46cd219df51359d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:42:32.103153 26215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65c274fde730484da46cd219df51359d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65c274fde730484da46cd219df51359d. Latest consensus state: current_term: 1 leader_uuid: "65c274fde730484da46cd219df51359d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c274fde730484da46cd219df51359d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 38755 } } }
I20260926 16:42:32.103132 26214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65c274fde730484da46cd219df51359d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65c274fde730484da46cd219df51359d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c274fde730484da46cd219df51359d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 38755 } } }
I20260926 16:42:32.103302 26214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65c274fde730484da46cd219df51359d [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:32.103300 26215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65c274fde730484da46cd219df51359d [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:32.103639 26222 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:42:32.106520 26222 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:42:32.111847 26222 catalog_manager.cc:1347] Generated new cluster ID: 08acc6f10e724bf1b7a388cfedf5a997
I20260926 16:42:32.111925 26222 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:42:32.136704 26222 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:42:32.137030 26222 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:42:32.143541 26222 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65c274fde730484da46cd219df51359d: Generated new TSK 0
I20260926 16:42:32.144191 26222 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:42:32.153908 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.65:0
--local_ip_for_outbound_sockets=127.21.12.65
--webserver_interface=127.21.12.65
--webserver_port=0
--tserver_master_addrs=127.21.12.126:38755
--builtin_ntp_servers=127.21.12.84:42541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 16:42:32.264864 26232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:32.265085 26232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:32.265123 26232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:32.268953 26232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:32.268992 26232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:32.269062 26232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.65
I20260926 16:42:32.273787 26232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:42541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.65
--webserver_port=0
--tserver_master_addrs=127.21.12.126:38755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.65
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:32.274649 26232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:32.275908 26232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:32.285231 26240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:32.285210 26237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:32.285259 26238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:32.285606 26232 server_base.cc:1034] running on GCE node
I20260926 16:42:32.286078 26232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:32.286660 26232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:32.287868 26232 hybrid_clock.cc:648] HybridClock initialized: now 1790440952287847 us; error 39 us; skew 500 ppm
I20260926 16:42:32.289677 26232 webserver.cc:460] Webserver started at http://127.21.12.65:43605/ using document root <none> and password file <none>
I20260926 16:42:32.290246 26232 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:32.290346 26232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:32.290563 26232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:32.292578 26232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-0/data/instance:
uuid: "894697cc41ee4ab38d3be2e723f5ccd4"
format_stamp: "Formatted at 2026-09-26 16:42:32 on dist-test-slave-n1hw"
server_key: "6d712aef26ed44d361e6518dc87ea5be"
server_key_iv: "3d34b08e0285f36f56cb22b0872e8119"
server_key_version: "encryptionkey@0"
I20260926 16:42:32.293123 26232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance:
uuid: "894697cc41ee4ab38d3be2e723f5ccd4"
format_stamp: "Formatted at 2026-09-26 16:42:32 on dist-test-slave-n1hw"
server_key: "6d712aef26ed44d361e6518dc87ea5be"
server_key_iv: "3d34b08e0285f36f56cb22b0872e8119"
server_key_version: "encryptionkey@0"
I20260926 16:42:32.296576 26232 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 16:42:32.299062 26246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:32.300441 26232 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 16:42:32.300583 26232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-0/wal
uuid: "894697cc41ee4ab38d3be2e723f5ccd4"
format_stamp: "Formatted at 2026-09-26 16:42:32 on dist-test-slave-n1hw"
server_key: "6d712aef26ed44d361e6518dc87ea5be"
server_key_iv: "3d34b08e0285f36f56cb22b0872e8119"
server_key_version: "encryptionkey@0"
I20260926 16:42:32.300720 26232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:32.306695 26232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:32.307451 26232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:32.307601 26232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:32.308099 26232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:42:32.308995 26232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:32.309062 26232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:32.309103 26232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:32.309120 26232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:32.318204 26232 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.65:41817
I20260926 16:42:32.318228 26359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.65:41817 every 8 connection(s)
I20260926 16:42:32.319116 26232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
I20260926 16:42:32.320312 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 26232
I20260926 16:42:32.320462 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance
I20260926 16:42:32.320811 21553 external_mini_cluster.cc:1428] Setting key 475b00c50cc76ef94bcc7ba7e2548f94
I20260926 16:42:32.323028 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.66:0
--local_ip_for_outbound_sockets=127.21.12.66
--webserver_interface=127.21.12.66
--webserver_port=0
--tserver_master_addrs=127.21.12.126:38755
--builtin_ntp_servers=127.21.12.84:42541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 16:42:32.326208 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:32.321005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:53993 (local address 127.21.12.126:38755)
0926 16:42:32.321315 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:32.321320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:32.322478 (+ 1158us) server_negotiation.cc:407] Connection header received
0926 16:42:32.324299 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:32.324303 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:32.324345 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:32.324442 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:32.325681 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:32.325685 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:32.325688 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:32.325703 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:32.325729 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:32.325738 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:32.325876 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:32.326031 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 16:42:32.326077 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209}
I20260926 16:42:32.327059 26362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:32.321514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38755 (local address 127.21.12.65:53993)
0926 16:42:32.322308 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:32.322373 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:32.324048 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:32.324662 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:32.324671 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:32.324765 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:32.325414 (+ 649us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:42:32.325415 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:32.325456 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:42:32.325525 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:32.325866 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:32.325876 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:32.325883 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 16:42:32.326075 (+ 192us) client_negotiation.cc:240] Negotiation successful
0926 16:42:32.326317 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":166,"threads_started":1}
I20260926 16:42:32.328362 26360 heartbeater.cc:347] Connected to a master server at 127.21.12.126:38755
I20260926 16:42:32.328595 26360 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:32.328977 26360 heartbeater.cc:511] Master 127.21.12.126:38755 requested a full tablet report, sending...
I20260926 16:42:32.330678 26172 ts_manager.cc:194] Registered new tserver with Master: 894697cc41ee4ab38d3be2e723f5ccd4 (127.21.12.65:41817)
W20260926 16:42:32.430740 26363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:32.430989 26363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:32.431057 26363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:32.434952 26363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:32.435026 26363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:32.435109 26363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.66
I20260926 16:42:32.439839 26363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:42541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.66
--webserver_port=0
--tserver_master_addrs=127.21.12.126:38755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.66
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:32.440788 26363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:32.442063 26363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:32.451108 26369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:32.451136 26371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:32.451332 26363 server_base.cc:1034] running on GCE node
W20260926 16:42:32.451109 26368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:32.451884 26363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:32.452472 26363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:32.453663 26363 hybrid_clock.cc:648] HybridClock initialized: now 1790440952453651 us; error 23 us; skew 500 ppm
I20260926 16:42:32.455446 26363 webserver.cc:460] Webserver started at http://127.21.12.66:36201/ using document root <none> and password file <none>
I20260926 16:42:32.456048 26363 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:32.456136 26363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:32.456349 26363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:32.458333 26363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-1/data/instance:
uuid: "910ff5e7d46749b183141544d8c9ec24"
format_stamp: "Formatted at 2026-09-26 16:42:32 on dist-test-slave-n1hw"
server_key: "7c7609d442ef29a4ed78d19dd957d81e"
server_key_iv: "d43692352a70a2f6c474ddb1c2144518"
server_key_version: "encryptionkey@0"
I20260926 16:42:32.458935 26363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance:
uuid: "910ff5e7d46749b183141544d8c9ec24"
format_stamp: "Formatted at 2026-09-26 16:42:32 on dist-test-slave-n1hw"
server_key: "7c7609d442ef29a4ed78d19dd957d81e"
server_key_iv: "d43692352a70a2f6c474ddb1c2144518"
server_key_version: "encryptionkey@0"
I20260926 16:42:32.462349 26363 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:42:32.464650 26377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:32.465845 26363 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:42:32.465969 26363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-1/wal
uuid: "910ff5e7d46749b183141544d8c9ec24"
format_stamp: "Formatted at 2026-09-26 16:42:32 on dist-test-slave-n1hw"
server_key: "7c7609d442ef29a4ed78d19dd957d81e"
server_key_iv: "d43692352a70a2f6c474ddb1c2144518"
server_key_version: "encryptionkey@0"
I20260926 16:42:32.466075 26363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:32.470764 26363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:32.471387 26363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:32.471565 26363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:32.472093 26363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:42:32.472937 26363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:32.473008 26363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:32.473070 26363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:32.473117 26363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:32.481458 26363 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.66:41903
I20260926 16:42:32.481480 26490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.66:41903 every 8 connection(s)
I20260926 16:42:32.482419 26363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
I20260926 16:42:32.488291 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:32.484180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:40009 (local address 127.21.12.126:38755)
0926 16:42:32.484364 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:32.484368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:32.485378 (+ 1010us) server_negotiation.cc:407] Connection header received
0926 16:42:32.486803 (+ 1425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:32.486806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:32.486842 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:32.486920 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:32.487917 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:32.487919 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:32.487921 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:32.487933 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:32.487954 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:32.487962 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:32.488038 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:32.488150 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 16:42:32.488187 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 16:42:32.489125 26493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:32.484555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38755 (local address 127.21.12.66:40009)
0926 16:42:32.485219 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:32.485270 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:32.486591 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:32.487076 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:32.487083 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:32.487171 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:32.487680 (+ 509us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:42:32.487681 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:32.487719 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:42:32.487782 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:32.488034 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:32.488040 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:32.488044 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 16:42:32.488168 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 16:42:32.488347 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":109,"threads_started":1}
I20260926 16:42:32.489532 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 26363
I20260926 16:42:32.489660 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance
I20260926 16:42:32.490013 21553 external_mini_cluster.cc:1428] Setting key 565c23fe68c5038ec752fbb7f37df234
I20260926 16:42:32.490392 26491 heartbeater.cc:347] Connected to a master server at 127.21.12.126:38755
I20260926 16:42:32.490623 26491 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:32.491050 26491 heartbeater.cc:511] Master 127.21.12.126:38755 requested a full tablet report, sending...
I20260926 16:42:32.492069 26172 ts_manager.cc:194] Registered new tserver with Master: 910ff5e7d46749b183141544d8c9ec24 (127.21.12.66:41903)
I20260926 16:42:32.492065 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.67:0
--local_ip_for_outbound_sockets=127.21.12.67
--webserver_interface=127.21.12.67
--webserver_port=0
--tserver_master_addrs=127.21.12.126:38755
--builtin_ntp_servers=127.21.12.84:42541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 16:42:32.597741 26494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:32.597980 26494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:32.598055 26494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:32.602108 26494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:32.602149 26494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:32.602241 26494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.67
I20260926 16:42:32.607329 26494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:42541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.67
--webserver_port=0
--tserver_master_addrs=127.21.12.126:38755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.67
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:32.608214 26494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:32.609472 26494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:32.618440 26502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:32.618427 26499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:32.618426 26500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:32.618841 26494 server_base.cc:1034] running on GCE node
I20260926 16:42:32.619207 26494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:32.619840 26494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:32.621023 26494 hybrid_clock.cc:648] HybridClock initialized: now 1790440952621011 us; error 31 us; skew 500 ppm
I20260926 16:42:32.622741 26494 webserver.cc:460] Webserver started at http://127.21.12.67:37555/ using document root <none> and password file <none>
I20260926 16:42:32.623282 26494 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:32.623370 26494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:32.623594 26494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:32.625473 26494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-2/data/instance:
uuid: "c9d7ffe781d749ee8abd879383f5fe84"
format_stamp: "Formatted at 2026-09-26 16:42:32 on dist-test-slave-n1hw"
server_key: "7c6a24fb2d6bd5e14000b68a956106fd"
server_key_iv: "fb81b13e762485feef809cff487fdace"
server_key_version: "encryptionkey@0"
I20260926 16:42:32.626016 26494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance:
uuid: "c9d7ffe781d749ee8abd879383f5fe84"
format_stamp: "Formatted at 2026-09-26 16:42:32 on dist-test-slave-n1hw"
server_key: "7c6a24fb2d6bd5e14000b68a956106fd"
server_key_iv: "fb81b13e762485feef809cff487fdace"
server_key_version: "encryptionkey@0"
I20260926 16:42:32.629456 26494 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260926 16:42:32.631843 26508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:32.633008 26494 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:42:32.633140 26494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-2/wal
uuid: "c9d7ffe781d749ee8abd879383f5fe84"
format_stamp: "Formatted at 2026-09-26 16:42:32 on dist-test-slave-n1hw"
server_key: "7c6a24fb2d6bd5e14000b68a956106fd"
server_key_iv: "fb81b13e762485feef809cff487fdace"
server_key_version: "encryptionkey@0"
I20260926 16:42:32.633245 26494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:32.637516 26494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:32.638111 26494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:32.638284 26494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:32.638784 26494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:42:32.639596 26494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:32.639663 26494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:32.639724 26494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:32.639773 26494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:32.648581 26494 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.67:40251
I20260926 16:42:32.648604 26621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.67:40251 every 8 connection(s)
I20260926 16:42:32.649406 26494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
I20260926 16:42:32.655149 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:32.651131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:33545 (local address 127.21.12.126:38755)
0926 16:42:32.651295 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:32.651300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:32.652296 (+ 996us) server_negotiation.cc:407] Connection header received
0926 16:42:32.653516 (+ 1220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:32.653518 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:32.653554 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:32.653628 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:32.654747 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:32.654750 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:32.654752 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:32.654764 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:32.654799 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:32.654807 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:32.654882 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:32.655016 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 16:42:32.655051 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 16:42:32.655977 26624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:32.651541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38755 (local address 127.21.12.67:33545)
0926 16:42:32.652154 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:32.652191 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:32.653327 (+ 1136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:32.653856 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:32.653862 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:32.653948 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:32.654474 (+ 526us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:42:32.654476 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:32.654535 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:42:32.654603 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:32.654891 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:32.654897 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:32.654902 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 16:42:32.655033 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 16:42:32.655244 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":107,"threads_started":1}
I20260926 16:42:32.657100 26622 heartbeater.cc:347] Connected to a master server at 127.21.12.126:38755
I20260926 16:42:32.657413 26622 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:32.657733 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 26494
I20260926 16:42:32.657922 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance
I20260926 16:42:32.657922 26622 heartbeater.cc:511] Master 127.21.12.126:38755 requested a full tablet report, sending...
I20260926 16:42:32.658232 21553 external_mini_cluster.cc:1428] Setting key 56400ed10741ffcb6a2a9ca0bf4b2cd7
I20260926 16:42:32.659029 26172 ts_manager.cc:194] Registered new tserver with Master: c9d7ffe781d749ee8abd879383f5fe84 (127.21.12.67:40251)
I20260926 16:42:32.660302 21553 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:42:32.663723 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:32.662249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37510 (local address 127.21.12.126:38755)
0926 16:42:32.662370 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:32.662374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:32.662590 (+ 216us) server_negotiation.cc:407] Connection header received
0926 16:42:32.662681 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:32.662683 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:32.662711 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:32.662782 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:32.663410 (+ 628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:32.663413 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:32.663415 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:32.663426 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:32.663442 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:32.663450 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:32.663522 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:32.663608 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 16:42:32.663638 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 16:42:32.666579 26172 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:42:32.669414 26172 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:42:32.680464 26637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:32.677998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57188 (local address 127.21.12.66:41903)
0926 16:42:32.678823 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:32.678836 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:32.678858 (+ 22us) server_negotiation.cc:407] Connection header received
0926 16:42:32.678910 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:32.678915 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:32.679100 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:32.679242 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:32.679868 (+ 626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:32.679874 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:32.679886 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:32.679896 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:32.679934 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:32.679973 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:32.680115 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:32.680255 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 16:42:32.680323 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":149,"threads_started":1}
I20260926 16:42:32.680428 26633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:32.677591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:41903 (local address 127.0.0.1:57188)
0926 16:42:32.678228 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:32.678243 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:32.678309 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:32.679224 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:32.679227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:32.679241 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:32.679737 (+ 496us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:42:32.679738 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:32.679747 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:42:32.679766 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:32.680109 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:32.680119 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:32.680126 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 16:42:32.680241 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 16:42:32.680311 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":3,"thread_start_us":79,"threads_started":1}
I20260926 16:42:32.680438 26635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:32.677358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:41817 (local address 127.0.0.1:34900)
0926 16:42:32.678012 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:32.678059 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:32.678158 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:32.678878 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:32.678883 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:32.678953 (+ 70us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:32.679513 (+ 560us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:42:32.679516 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:32.679537 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:42:32.679608 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:32.680130 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:32.680136 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:32.680140 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 16:42:32.680234 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 16:42:32.680304 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":79,"threads_started":1}
I20260926 16:42:32.680810 26634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:32.677494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34900 (local address 127.21.12.65:41817)
0926 16:42:32.678419 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:32.678433 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:32.678472 (+ 39us) server_negotiation.cc:407] Connection header received
0926 16:42:32.678525 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:32.678530 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:32.678720 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:32.678869 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:32.679900 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:32.679905 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:32.679913 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:32.679923 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:32.679955 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:32.679976 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:32.680085 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:32.680574 (+ 489us) server_negotiation.cc:299] Negotiation successful
0926 16:42:32.680645 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":813,"thread_start_us":112,"threads_started":1}
I20260926 16:42:32.682392 26638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:32.678288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.67:40251 (local address 127.0.0.1:59944)
0926 16:42:32.678670 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:32.678683 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:32.678759 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:32.679863 (+ 1104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:32.679867 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:32.679885 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:32.680642 (+ 757us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:42:32.680644 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:32.680654 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:42:32.680679 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:32.682133 (+ 1454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:32.682135 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:32.682138 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 16:42:32.682250 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 16:42:32.682284 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":102,"threads_started":1}
I20260926 16:42:32.682461 26636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:32.677974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59944 (local address 127.21.12.67:40251)
0926 16:42:32.678682 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:32.678695 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:32.678749 (+ 54us) server_negotiation.cc:407] Connection header received
0926 16:42:32.678905 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:32.678911 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:32.679075 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:32.679200 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:32.681663 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:32.681669 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:32.681680 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:32.681694 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:32.681951 (+ 257us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:32.681988 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:32.682090 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:32.682267 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 16:42:32.682329 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":122,"threads_started":1}
I20260926 16:42:32.683533 26425 tablet_service.cc:1467] Processing CreateTablet for tablet 7314a60414fe49d6b7fe339a5b876477 (DEFAULT_TABLE table=test-table [id=fbda68d44c9c41559054c7049253489b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:32.683537 26294 tablet_service.cc:1467] Processing CreateTablet for tablet 7314a60414fe49d6b7fe339a5b876477 (DEFAULT_TABLE table=test-table [id=fbda68d44c9c41559054c7049253489b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:32.684600 26425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7314a60414fe49d6b7fe339a5b876477. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:32.684600 26294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7314a60414fe49d6b7fe339a5b876477. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:32.684824 26556 tablet_service.cc:1467] Processing CreateTablet for tablet 7314a60414fe49d6b7fe339a5b876477 (DEFAULT_TABLE table=test-table [id=fbda68d44c9c41559054c7049253489b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:32.685822 26556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7314a60414fe49d6b7fe339a5b876477. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:32.691744 26640 tablet_bootstrap.cc:492] T 7314a60414fe49d6b7fe339a5b876477 P 910ff5e7d46749b183141544d8c9ec24: Bootstrap starting.
I20260926 16:42:32.692193 26639 tablet_bootstrap.cc:492] T 7314a60414fe49d6b7fe339a5b876477 P 894697cc41ee4ab38d3be2e723f5ccd4: Bootstrap starting.
I20260926 16:42:32.692826 26641 tablet_bootstrap.cc:492] T 7314a60414fe49d6b7fe339a5b876477 P c9d7ffe781d749ee8abd879383f5fe84: Bootstrap starting.
I20260926 16:42:32.694756 26639 tablet_bootstrap.cc:654] T 7314a60414fe49d6b7fe339a5b876477 P 894697cc41ee4ab38d3be2e723f5ccd4: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:32.695096 26641 tablet_bootstrap.cc:654] T 7314a60414fe49d6b7fe339a5b876477 P c9d7ffe781d749ee8abd879383f5fe84: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:32.695808 26639 log.cc:824] T 7314a60414fe49d6b7fe339a5b876477 P 894697cc41ee4ab38d3be2e723f5ccd4: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:32.695892 26641 log.cc:824] T 7314a60414fe49d6b7fe339a5b876477 P c9d7ffe781d749ee8abd879383f5fe84: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:32.697857 26641 tablet_bootstrap.cc:492] T 7314a60414fe49d6b7fe339a5b876477 P c9d7ffe781d749ee8abd879383f5fe84: No bootstrap required, opened a new log
I20260926 16:42:32.697983 26639 tablet_bootstrap.cc:492] T 7314a60414fe49d6b7fe339a5b876477 P 894697cc41ee4ab38d3be2e723f5ccd4: No bootstrap required, opened a new log
I20260926 16:42:32.698094 26641 ts_tablet_manager.cc:1397] T 7314a60414fe49d6b7fe339a5b876477 P c9d7ffe781d749ee8abd879383f5fe84: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20260926 16:42:32.698173 26639 ts_tablet_manager.cc:1397] T 7314a60414fe49d6b7fe339a5b876477 P 894697cc41ee4ab38d3be2e723f5ccd4: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 16:42:32.698436 26640 tablet_bootstrap.cc:654] T 7314a60414fe49d6b7fe339a5b876477 P 910ff5e7d46749b183141544d8c9ec24: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:32.699285 26640 log.cc:824] T 7314a60414fe49d6b7fe339a5b876477 P 910ff5e7d46749b183141544d8c9ec24: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:32.701277 26640 tablet_bootstrap.cc:492] T 7314a60414fe49d6b7fe339a5b876477 P 910ff5e7d46749b183141544d8c9ec24: No bootstrap required, opened a new log
I20260926 16:42:32.701505 26640 ts_tablet_manager.cc:1397] T 7314a60414fe49d6b7fe339a5b876477 P 910ff5e7d46749b183141544d8c9ec24: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260926 16:42:32.701949 26641 raft_consensus.cc:348] T 7314a60414fe49d6b7fe339a5b876477 P c9d7ffe781d749ee8abd879383f5fe84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894697cc41ee4ab38d3be2e723f5ccd4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41817 } } peers { permanent_uuid: "910ff5e7d46749b183141544d8c9ec24" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 41903 } } peers { permanent_uuid: "c9d7ffe781d749ee8abd879383f5fe84" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 40251 } }
I20260926 16:42:32.702198 26641 raft_consensus.cc:374] T 7314a60414fe49d6b7fe339a5b876477 P c9d7ffe781d749ee8abd879383f5fe84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:32.702262 26641 raft_consensus.cc:729] T 7314a60414fe49d6b7fe339a5b876477 P c9d7ffe781d749ee8abd879383f5fe84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9d7ffe781d749ee8abd879383f5fe84, State: Initialized, Role: FOLLOWER
I20260926 16:42:32.702688 26641 consensus_queue.cc:260] T 7314a60414fe49d6b7fe339a5b876477 P c9d7ffe781d749ee8abd879383f5fe84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894697cc41ee4ab38d3be2e723f5ccd4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41817 } } peers { permanent_uuid: "910ff5e7d46749b183141544d8c9ec24" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 41903 } } peers { permanent_uuid: "c9d7ffe781d749ee8abd879383f5fe84" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 40251 } }
I20260926 16:42:32.703348 26622 heartbeater.cc:503] Master 127.21.12.126:38755 was elected leader, sending a full tablet report...
I20260926 16:42:32.703616 26641 ts_tablet_manager.cc:1428] T 7314a60414fe49d6b7fe339a5b876477 P c9d7ffe781d749ee8abd879383f5fe84: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 16:42:32.703097 26639 raft_consensus.cc:348] T 7314a60414fe49d6b7fe339a5b876477 P 894697cc41ee4ab38d3be2e723f5ccd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894697cc41ee4ab38d3be2e723f5ccd4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41817 } } peers { permanent_uuid: "910ff5e7d46749b183141544d8c9ec24" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 41903 } } peers { permanent_uuid: "c9d7ffe781d749ee8abd879383f5fe84" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 40251 } }
I20260926 16:42:32.704651 26639 raft_consensus.cc:374] T 7314a60414fe49d6b7fe339a5b876477 P 894697cc41ee4ab38d3be2e723f5ccd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:32.704736 26639 raft_consensus.cc:729] T 7314a60414fe49d6b7fe339a5b876477 P 894697cc41ee4ab38d3be2e723f5ccd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 894697cc41ee4ab38d3be2e723f5ccd4, State: Initialized, Role: FOLLOWER
I20260926 16:42:32.705255 26639 consensus_queue.cc:260] T 7314a60414fe49d6b7fe339a5b876477 P 894697cc41ee4ab38d3be2e723f5ccd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894697cc41ee4ab38d3be2e723f5ccd4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41817 } } peers { permanent_uuid: "910ff5e7d46749b183141544d8c9ec24" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 41903 } } peers { permanent_uuid: "c9d7ffe781d749ee8abd879383f5fe84" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 40251 } }
I20260926 16:42:32.705946 26360 heartbeater.cc:503] Master 127.21.12.126:38755 was elected leader, sending a full tablet report...
I20260926 16:42:32.706424 26640 raft_consensus.cc:348] T 7314a60414fe49d6b7fe339a5b876477 P 910ff5e7d46749b183141544d8c9ec24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894697cc41ee4ab38d3be2e723f5ccd4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41817 } } peers { permanent_uuid: "910ff5e7d46749b183141544d8c9ec24" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 41903 } } peers { permanent_uuid: "c9d7ffe781d749ee8abd879383f5fe84" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 40251 } }
I20260926 16:42:32.706670 26640 raft_consensus.cc:374] T 7314a60414fe49d6b7fe339a5b876477 P 910ff5e7d46749b183141544d8c9ec24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:32.706733 26640 raft_consensus.cc:729] T 7314a60414fe49d6b7fe339a5b876477 P 910ff5e7d46749b183141544d8c9ec24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 910ff5e7d46749b183141544d8c9ec24, State: Initialized, Role: FOLLOWER
I20260926 16:42:32.706863 26639 ts_tablet_manager.cc:1428] T 7314a60414fe49d6b7fe339a5b876477 P 894697cc41ee4ab38d3be2e723f5ccd4: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260926 16:42:32.707208 26640 consensus_queue.cc:260] T 7314a60414fe49d6b7fe339a5b876477 P 910ff5e7d46749b183141544d8c9ec24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894697cc41ee4ab38d3be2e723f5ccd4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41817 } } peers { permanent_uuid: "910ff5e7d46749b183141544d8c9ec24" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 41903 } } peers { permanent_uuid: "c9d7ffe781d749ee8abd879383f5fe84" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 40251 } }
I20260926 16:42:32.707862 26491 heartbeater.cc:503] Master 127.21.12.126:38755 was elected leader, sending a full tablet report...
I20260926 16:42:32.708227 26640 ts_tablet_manager.cc:1428] T 7314a60414fe49d6b7fe339a5b876477 P 910ff5e7d46749b183141544d8c9ec24: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260926 16:42:32.733309 26492 tablet.cc:2369] T 7314a60414fe49d6b7fe339a5b876477 P 910ff5e7d46749b183141544d8c9ec24: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:42:32.736927 26645 raft_consensus.cc:482] T 7314a60414fe49d6b7fe339a5b876477 P c9d7ffe781d749ee8abd879383f5fe84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:42:32.737166 26645 raft_consensus.cc:504] T 7314a60414fe49d6b7fe339a5b876477 P c9d7ffe781d749ee8abd879383f5fe84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894697cc41ee4ab38d3be2e723f5ccd4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41817 } } peers { permanent_uuid: "910ff5e7d46749b183141544d8c9ec24" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 41903 } } peers { permanent_uuid: "c9d7ffe781d749ee8abd879383f5fe84" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 40251 } }
I20260926 16:42:32.738407 26645 leader_election.cc:288] T 7314a60414fe49d6b7fe339a5b876477 P c9d7ffe781d749ee8abd879383f5fe84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 894697cc41ee4ab38d3be2e723f5ccd4 (127.21.12.65:41817), 910ff5e7d46749b183141544d8c9ec24 (127.21.12.66:41903)
I20260926 16:42:32.740504 26637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:32.738861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:43725 (local address 127.21.12.66:41903)
0926 16:42:32.738983 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:32.738987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:32.739209 (+ 222us) server_negotiation.cc:407] Connection header received
0926 16:42:32.739264 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:32.739266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:32.739306 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:32.739366 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:32.740057 (+ 691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:32.740061 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:32.740063 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:32.740082 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:32.740100 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:32.740109 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:32.740242 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:32.740317 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 16:42:32.740363 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 16:42:32.740533 26648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:32.738691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:41903 (local address 127.21.12.67:43725)
0926 16:42:32.739079 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:32.739091 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:32.739197 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:32.739418 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:32.739422 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:32.739440 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:32.739880 (+ 440us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:42:32.739882 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:32.739894 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:42:32.739929 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:32.740245 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:32.740248 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:32.740250 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 16:42:32.740312 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 16:42:32.740367 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":106,"threads_started":1}
I20260926 16:42:32.740561 26624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:32.738669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:41817 (local address 127.21.12.67:50009)
0926 16:42:32.738850 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:32.738866 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:32.738956 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:32.739202 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:32.739205 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:32.739222 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:32.740080 (+ 858us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:42:32.740081 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:32.740089 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:42:32.740106 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:32.740373 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:32.740375 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:32.740378 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 16:42:32.740448 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 16:42:32.740474 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260926 16:42:32.740636 26634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:32.738773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:50009 (local address 127.21.12.65:41817)
0926 16:42:32.738897 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:32.738903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:32.738921 (+ 18us) server_negotiation.cc:407] Connection header received
0926 16:42:32.739054 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:32.739056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:32.739088 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:32.739159 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:32.740231 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:32.740234 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:32.740237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:32.740250 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:32.740273 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:32.740285 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:32.740371 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:32.740478 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 16:42:32.740517 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 16:42:32.741262 26445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7314a60414fe49d6b7fe339a5b876477" candidate_uuid: "c9d7ffe781d749ee8abd879383f5fe84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "910ff5e7d46749b183141544d8c9ec24" is_pre_election: true
I20260926 16:42:32.741261 26314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7314a60414fe49d6b7fe339a5b876477" candidate_uuid: "c9d7ffe781d749ee8abd879383f5fe84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "894697cc41ee4ab38d3be2e723f5ccd4" is_pre_election: true
I20260926 16:42:32.741557 26445 raft_consensus.cc:2454] T 7314a60414fe49d6b7fe339a5b876477 P 910ff5e7d46749b183141544d8c9ec24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9d7ffe781d749ee8abd879383f5fe84 in term 0.
I20260926 16:42:32.741557 26314 raft_consensus.cc:2454] T 7314a60414fe49d6b7fe339a5b876477 P 894697cc41ee4ab38d3be2e723f5ccd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9d7ffe781d749ee8abd879383f5fe84 in term 0.
I20260926 16:42:32.742028 26509 leader_election.cc:302] T 7314a60414fe49d6b7fe339a5b876477 P c9d7ffe781d749ee8abd879383f5fe84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 894697cc41ee4ab38d3be2e723f5ccd4, 910ff5e7d46749b183141544d8c9ec24, c9d7ffe781d749ee8abd879383f5fe84; no voters:
I20260926 16:42:32.742285 26645 raft_consensus.cc:2781] T 7314a60414fe49d6b7fe339a5b876477 P c9d7ffe781d749ee8abd879383f5fe84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:42:32.742384 26645 raft_consensus.cc:482] T 7314a60414fe49d6b7fe339a5b876477 P c9d7ffe781d749ee8abd879383f5fe84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:42:32.742451 26645 raft_consensus.cc:3037] T 7314a60414fe49d6b7fe339a5b876477 P c9d7ffe781d749ee8abd879383f5fe84 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:32.743476 26645 raft_consensus.cc:504] T 7314a60414fe49d6b7fe339a5b876477 P c9d7ffe781d749ee8abd879383f5fe84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894697cc41ee4ab38d3be2e723f5ccd4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41817 } } peers { permanent_uuid: "910ff5e7d46749b183141544d8c9ec24" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 41903 } } peers { permanent_uuid: "c9d7ffe781d749ee8abd879383f5fe84" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 40251 } }
I20260926 16:42:32.743805 26645 leader_election.cc:288] T 7314a60414fe49d6b7fe339a5b876477 P c9d7ffe781d749ee8abd879383f5fe84 [CANDIDATE]: Term 1 election: Requested vote from peers 894697cc41ee4ab38d3be2e723f5ccd4 (127.21.12.65:41817), 910ff5e7d46749b183141544d8c9ec24 (127.21.12.66:41903)
I20260926 16:42:32.744154 26314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7314a60414fe49d6b7fe339a5b876477" candidate_uuid: "c9d7ffe781d749ee8abd879383f5fe84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "894697cc41ee4ab38d3be2e723f5ccd4"
I20260926 16:42:32.744177 26445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7314a60414fe49d6b7fe339a5b876477" candidate_uuid: "c9d7ffe781d749ee8abd879383f5fe84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "910ff5e7d46749b183141544d8c9ec24"
I20260926 16:42:32.744279 26445 raft_consensus.cc:3037] T 7314a60414fe49d6b7fe339a5b876477 P 910ff5e7d46749b183141544d8c9ec24 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:32.744280 26314 raft_consensus.cc:3037] T 7314a60414fe49d6b7fe339a5b876477 P 894697cc41ee4ab38d3be2e723f5ccd4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:32.745508 26314 raft_consensus.cc:2454] T 7314a60414fe49d6b7fe339a5b876477 P 894697cc41ee4ab38d3be2e723f5ccd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9d7ffe781d749ee8abd879383f5fe84 in term 1.
I20260926 16:42:32.745508 26445 raft_consensus.cc:2454] T 7314a60414fe49d6b7fe339a5b876477 P 910ff5e7d46749b183141544d8c9ec24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9d7ffe781d749ee8abd879383f5fe84 in term 1.
I20260926 16:42:32.745879 26509 leader_election.cc:302] T 7314a60414fe49d6b7fe339a5b876477 P c9d7ffe781d749ee8abd879383f5fe84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910ff5e7d46749b183141544d8c9ec24, c9d7ffe781d749ee8abd879383f5fe84; no voters:
I20260926 16:42:32.746109 26645 raft_consensus.cc:2781] T 7314a60414fe49d6b7fe339a5b876477 P c9d7ffe781d749ee8abd879383f5fe84 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:32.746352 26645 raft_consensus.cc:686] T 7314a60414fe49d6b7fe339a5b876477 P c9d7ffe781d749ee8abd879383f5fe84 [term 1 LEADER]: Becoming Leader. State: Replica: c9d7ffe781d749ee8abd879383f5fe84, State: Running, Role: LEADER
I20260926 16:42:32.746688 26645 consensus_queue.cc:237] T 7314a60414fe49d6b7fe339a5b876477 P c9d7ffe781d749ee8abd879383f5fe84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894697cc41ee4ab38d3be2e723f5ccd4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41817 } } peers { permanent_uuid: "910ff5e7d46749b183141544d8c9ec24" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 41903 } } peers { permanent_uuid: "c9d7ffe781d749ee8abd879383f5fe84" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 40251 } }
I20260926 16:42:32.750015 26172 catalog_manager.cc:5495] T 7314a60414fe49d6b7fe339a5b876477 P c9d7ffe781d749ee8abd879383f5fe84 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9d7ffe781d749ee8abd879383f5fe84 (127.21.12.67). New cstate: current_term: 1 leader_uuid: "c9d7ffe781d749ee8abd879383f5fe84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894697cc41ee4ab38d3be2e723f5ccd4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910ff5e7d46749b183141544d8c9ec24" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 41903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9d7ffe781d749ee8abd879383f5fe84" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 40251 } health_report { overall_health: HEALTHY } } }
I20260926 16:42:32.758996 26636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:32.757461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59958 (local address 127.21.12.67:40251)
0926 16:42:32.757579 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:32.757583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:32.757597 (+ 14us) server_negotiation.cc:407] Connection header received
0926 16:42:32.757724 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:32.757726 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:32.757829 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:32.757915 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:32.758618 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:32.758621 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:32.758624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:32.758635 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:32.758653 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:32.758662 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:32.758757 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:32.758839 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 16:42:32.758878 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 16:42:32.764608 26445 raft_consensus.cc:1261] T 7314a60414fe49d6b7fe339a5b876477 P 910ff5e7d46749b183141544d8c9ec24 [term 1 FOLLOWER]: Refusing update from remote peer c9d7ffe781d749ee8abd879383f5fe84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:42:32.764607 26314 raft_consensus.cc:1261] T 7314a60414fe49d6b7fe339a5b876477 P 894697cc41ee4ab38d3be2e723f5ccd4 [term 1 FOLLOWER]: Refusing update from remote peer c9d7ffe781d749ee8abd879383f5fe84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:42:32.765268 26649 consensus_queue.cc:1035] T 7314a60414fe49d6b7fe339a5b876477 P c9d7ffe781d749ee8abd879383f5fe84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "894697cc41ee4ab38d3be2e723f5ccd4" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:42:32.765451 26645 consensus_queue.cc:1035] T 7314a60414fe49d6b7fe339a5b876477 P c9d7ffe781d749ee8abd879383f5fe84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910ff5e7d46749b183141544d8c9ec24" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 41903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:42:32.773808 26654 mvcc.cc:205] Tried to move back new op lower bound from 7333646142517338112 to 7333646142453432320. Current Snapshot: MvccSnapshot[applied={T|T < 7333646142517338112}]
I20260926 16:42:32.780071 26171 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:42:32.780233 26171 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:42:32.783972 26171 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 65c274fde730484da46cd219df51359d: Sending DeleteTablet for 3 replicas of tablet 7314a60414fe49d6b7fe339a5b876477
I20260926 16:42:32.784726 26294 tablet_service.cc:1514] Processing DeleteTablet for tablet 7314a60414fe49d6b7fe339a5b876477 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:32 UTC) from {username='slave'} at 127.0.0.1:34900
I20260926 16:42:32.784812 26425 tablet_service.cc:1514] Processing DeleteTablet for tablet 7314a60414fe49d6b7fe339a5b876477 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:32 UTC) from {username='slave'} at 127.0.0.1:57188
I20260926 16:42:32.785228 26556 tablet_service.cc:1514] Processing DeleteTablet for tablet 7314a60414fe49d6b7fe339a5b876477 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:32 UTC) from {username='slave'} at 127.0.0.1:59944
I20260926 16:42:32.785301 26664 tablet_replica.cc:330] T 7314a60414fe49d6b7fe339a5b876477 P 910ff5e7d46749b183141544d8c9ec24: stopping tablet replica
I20260926 16:42:32.785640 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 26232
I20260926 16:42:32.785655 26665 tablet_replica.cc:330] T 7314a60414fe49d6b7fe339a5b876477 P c9d7ffe781d749ee8abd879383f5fe84: stopping tablet replica
I20260926 16:42:32.785687 26664 raft_consensus.cc:2229] T 7314a60414fe49d6b7fe339a5b876477 P 910ff5e7d46749b183141544d8c9ec24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:42:32.786412 26664 raft_consensus.cc:2258] T 7314a60414fe49d6b7fe339a5b876477 P 910ff5e7d46749b183141544d8c9ec24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:42:32.787093 26665 raft_consensus.cc:2229] T 7314a60414fe49d6b7fe339a5b876477 P c9d7ffe781d749ee8abd879383f5fe84 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:42:32.787485 26665 raft_consensus.cc:2258] T 7314a60414fe49d6b7fe339a5b876477 P c9d7ffe781d749ee8abd879383f5fe84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:42:32.788050 26664 ts_tablet_manager.cc:1905] T 7314a60414fe49d6b7fe339a5b876477 P 910ff5e7d46749b183141544d8c9ec24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:42:32.788833 26665 ts_tablet_manager.cc:1905] T 7314a60414fe49d6b7fe339a5b876477 P c9d7ffe781d749ee8abd879383f5fe84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:42:32.790930 26664 ts_tablet_manager.cc:1918] T 7314a60414fe49d6b7fe339a5b876477 P 910ff5e7d46749b183141544d8c9ec24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:42:32.791100 26664 log.cc:1196] T 7314a60414fe49d6b7fe339a5b876477 P 910ff5e7d46749b183141544d8c9ec24: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-1/wal/wals/7314a60414fe49d6b7fe339a5b876477
I20260926 16:42:32.791494 26664 ts_tablet_manager.cc:1939] T 7314a60414fe49d6b7fe339a5b876477 P 910ff5e7d46749b183141544d8c9ec24: Deleting consensus metadata
I20260926 16:42:32.791576 26665 ts_tablet_manager.cc:1918] T 7314a60414fe49d6b7fe339a5b876477 P c9d7ffe781d749ee8abd879383f5fe84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:42:32.791663 26665 log.cc:1196] T 7314a60414fe49d6b7fe339a5b876477 P c9d7ffe781d749ee8abd879383f5fe84: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440924937351-21553-0/minicluster-data/ts-2/wal/wals/7314a60414fe49d6b7fe339a5b876477
I20260926 16:42:32.792023 26665 ts_tablet_manager.cc:1939] T 7314a60414fe49d6b7fe339a5b876477 P c9d7ffe781d749ee8abd879383f5fe84: Deleting consensus metadata
I20260926 16:42:32.792948 26158 catalog_manager.cc:4841] TS c9d7ffe781d749ee8abd879383f5fe84 (127.21.12.67:40251): tablet 7314a60414fe49d6b7fe339a5b876477 (table test-table [id=fbda68d44c9c41559054c7049253489b]) successfully deleted
I20260926 16:42:32.793097 26156 catalog_manager.cc:4841] TS 910ff5e7d46749b183141544d8c9ec24 (127.21.12.66:41903): tablet 7314a60414fe49d6b7fe339a5b876477 (table test-table [id=fbda68d44c9c41559054c7049253489b]) successfully deleted
I20260926 16:42:32.796110 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 26363
W20260926 16:42:32.796007 26159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.12.65:41817 (error 108)
I20260926 16:42:32.799847 26638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:32.799496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:41817 (local address 127.0.0.1:34910)
0926 16:42:32.799657 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:32.799741 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.65:41817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 16:42:32.800091 26159 catalog_manager.cc:4584] TS 894697cc41ee4ab38d3be2e723f5ccd4 (127.21.12.65:41817): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7314a60414fe49d6b7fe339a5b876477: Network error: Client connection negotiation failed: client connection to 127.21.12.65:41817: connect: Connection refused (error 111)
I20260926 16:42:32.803596 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 26494
I20260926 16:42:32.808750 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 26141
2026-09-26T16:42:32Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (912 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 16:42:32.830842 21553 mini_oidc.cc:152] Starting JWKS server
I20260926 16:42:32.832458 21553 webserver.cc:460] Webserver started at https://127.0.0.1:36931/ using document root <none> and password file <none>
I20260926 16:42:32.832598 21553 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 16:42:32.832830 21553 webserver.cc:460] Webserver started at http://127.0.0.1:42311/ using document root <none> and password file <none>
2026-09-26T16:42:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:42:32Z Disabled control of system clock
I20260926 16:42:32.842090 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:33101
--webserver_interface=127.21.12.126
--webserver_port=0
--builtin_ntp_servers=127.21.12.84:43641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:33101
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36931/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440924937351-21553-0/testchainca.cert with env {}
W20260926 16:42:32.950964 26672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:32.951196 26672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:32.951234 26672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:32.955062 26672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:42:32.955116 26672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:32.955130 26672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:32.955147 26672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:42:32.955161 26672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 16:42:32.955178 26672 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 16:42:32.955190 26672 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36931/jwks.json
I20260926 16:42:32.960377 26672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:43641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:33101
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:33101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36931/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440924937351-21553-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:32.961367 26672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:32.962340 26672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:32.970623 26680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:32.970690 26672 server_base.cc:1034] running on GCE node
W20260926 16:42:32.970606 26678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:32.970605 26677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:32.971333 26672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:32.972409 26672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:32.973573 26672 hybrid_clock.cc:648] HybridClock initialized: now 1790440952973559 us; error 25 us; skew 500 ppm
I20260926 16:42:32.975337 26672 webserver.cc:460] Webserver started at http://127.21.12.126:46617/ using document root <none> and password file <none>
I20260926 16:42:32.975934 26672 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:32.976017 26672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:32.976269 26672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:32.978300 26672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "977d9441b3174976a3fc087bd912ea52"
format_stamp: "Formatted at 2026-09-26 16:42:32 on dist-test-slave-n1hw"
server_key: "e016d4a4d6704c01e5d13841cccecc29"
server_key_iv: "63d8f066ed7f8e18e2295a8aa5e72640"
server_key_version: "encryptionkey@0"
I20260926 16:42:32.978852 26672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "977d9441b3174976a3fc087bd912ea52"
format_stamp: "Formatted at 2026-09-26 16:42:32 on dist-test-slave-n1hw"
server_key: "e016d4a4d6704c01e5d13841cccecc29"
server_key_iv: "63d8f066ed7f8e18e2295a8aa5e72640"
server_key_version: "encryptionkey@0"
I20260926 16:42:32.982378 26672 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:42:32.984669 26686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:32.986227 26672 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:42:32.986331 26672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "977d9441b3174976a3fc087bd912ea52"
format_stamp: "Formatted at 2026-09-26 16:42:32 on dist-test-slave-n1hw"
server_key: "e016d4a4d6704c01e5d13841cccecc29"
server_key_iv: "63d8f066ed7f8e18e2295a8aa5e72640"
server_key_version: "encryptionkey@0"
I20260926 16:42:32.986466 26672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:33.002942 26672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:33.003489 26672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:33.003624 26672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:33.009680 26672 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.126:33101
I20260926 16:42:33.009735 26738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.126:33101 every 8 connection(s)
I20260926 16:42:33.010603 26672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:42:33.013162 26739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:33.017398 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 26672
I20260926 16:42:33.017494 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:42:33.017817 21553 external_mini_cluster.cc:1428] Setting key ca3cfe8efc5a662bcffb126be6e4e603
I20260926 16:42:33.018479 26739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 977d9441b3174976a3fc087bd912ea52: Bootstrap starting.
I20260926 16:42:33.020571 26739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 977d9441b3174976a3fc087bd912ea52: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:33.021412 26739 log.cc:824] T 00000000000000000000000000000000 P 977d9441b3174976a3fc087bd912ea52: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:33.023478 26739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 977d9441b3174976a3fc087bd912ea52: No bootstrap required, opened a new log
I20260926 16:42:33.026887 26742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:33.019136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50828 (local address 127.21.12.126:33101)
0926 16:42:33.019743 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:33.019759 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:33.019792 (+ 33us) server_negotiation.cc:407] Connection header received
0926 16:42:33.020381 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:33.020397 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:33.020805 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:33.021074 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:33.021398 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:33.022579 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:33.023754 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:33.024208 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:33.024363 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:33.025175 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:33.025180 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:33.025192 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:33.025580 (+ 388us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:33.025631 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:33.025650 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:33.025723 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:33.025910 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 16:42:33.026181 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":162,"threads_started":1}
I20260926 16:42:33.027359 26739 raft_consensus.cc:348] T 00000000000000000000000000000000 P 977d9441b3174976a3fc087bd912ea52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977d9441b3174976a3fc087bd912ea52" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 33101 } }
I20260926 16:42:33.027597 26739 raft_consensus.cc:374] T 00000000000000000000000000000000 P 977d9441b3174976a3fc087bd912ea52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:33.027674 26739 raft_consensus.cc:729] T 00000000000000000000000000000000 P 977d9441b3174976a3fc087bd912ea52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 977d9441b3174976a3fc087bd912ea52, State: Initialized, Role: FOLLOWER
I20260926 16:42:33.028162 26739 consensus_queue.cc:260] T 00000000000000000000000000000000 P 977d9441b3174976a3fc087bd912ea52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977d9441b3174976a3fc087bd912ea52" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 33101 } }
I20260926 16:42:33.028314 26739 raft_consensus.cc:388] T 00000000000000000000000000000000 P 977d9441b3174976a3fc087bd912ea52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:42:33.028365 26739 raft_consensus.cc:482] T 00000000000000000000000000000000 P 977d9441b3174976a3fc087bd912ea52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:42:33.028456 26739 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 977d9441b3174976a3fc087bd912ea52 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:33.029548 26739 raft_consensus.cc:504] T 00000000000000000000000000000000 P 977d9441b3174976a3fc087bd912ea52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977d9441b3174976a3fc087bd912ea52" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 33101 } }
I20260926 16:42:33.029811 21553 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:42:33.029953 26739 leader_election.cc:302] T 00000000000000000000000000000000 P 977d9441b3174976a3fc087bd912ea52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 977d9441b3174976a3fc087bd912ea52; no voters:
I20260926 16:42:33.030287 26739 leader_election.cc:288] T 00000000000000000000000000000000 P 977d9441b3174976a3fc087bd912ea52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:42:33.030359 26744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 977d9441b3174976a3fc087bd912ea52 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:33.030539 26744 raft_consensus.cc:686] T 00000000000000000000000000000000 P 977d9441b3174976a3fc087bd912ea52 [term 1 LEADER]: Becoming Leader. State: Replica: 977d9441b3174976a3fc087bd912ea52, State: Running, Role: LEADER
I20260926 16:42:33.030817 26744 consensus_queue.cc:237] T 00000000000000000000000000000000 P 977d9441b3174976a3fc087bd912ea52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977d9441b3174976a3fc087bd912ea52" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 33101 } }
I20260926 16:42:33.031478 26739 sys_catalog.cc:564] T 00000000000000000000000000000000 P 977d9441b3174976a3fc087bd912ea52 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:42:33.032582 26746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 977d9441b3174976a3fc087bd912ea52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "977d9441b3174976a3fc087bd912ea52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977d9441b3174976a3fc087bd912ea52" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 33101 } } }
I20260926 16:42:33.032709 26746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 977d9441b3174976a3fc087bd912ea52 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:33.032965 26745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 977d9441b3174976a3fc087bd912ea52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 977d9441b3174976a3fc087bd912ea52. Latest consensus state: current_term: 1 leader_uuid: "977d9441b3174976a3fc087bd912ea52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977d9441b3174976a3fc087bd912ea52" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 33101 } } }
I20260926 16:42:33.033069 26745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 977d9441b3174976a3fc087bd912ea52 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:33.033164 26753 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:42:33.036216 26753 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:42:33.041458 26753 catalog_manager.cc:1347] Generated new cluster ID: 333d91dcad94484a942cf5b0bc9e9a42
I20260926 16:42:33.041520 26753 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:42:33.058296 26753 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:42:33.059228 26753 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:42:33.065550 26753 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 977d9441b3174976a3fc087bd912ea52: Generated new TSK 0
I20260926 16:42:33.066203 26753 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 16:42:33.093684 26769 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 16:42:33.093816 26769 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 16:42:33.094621 26742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:33.077900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50844 (local address 127.21.12.126:33101)
0926 16:42:33.078065 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:33.078071 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:33.078234 (+ 163us) server_negotiation.cc:407] Connection header received
0926 16:42:33.078430 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:33.078432 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:33.078481 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:33.078567 (+ 86us) server_negotiation.cc:226] Negotiated authn=JWT
0926 16:42:33.078828 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:33.079375 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:33.080566 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:33.080780 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:33.080864 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:33.081195 (+ 331us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 16:42:33.093990 (+ 12795us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36931/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 16:42:33.094177 (+ 187us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50844: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36931/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":79}
W20260926 16:42:33.094709 26742 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50844: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36931/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 16:42:33.095983 26768 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:33.077811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:33101 (local address 127.0.0.1:50844)
0926 16:42:33.078146 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:33.078161 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:33.078287 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:33.078604 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:33.078607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:33.078615 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0926 16:42:33.078725 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:33.078731 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:33.079476 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:33.079479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:33.080445 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:33.080454 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:33.080895 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:33.080899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:33.080985 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:33.081034 (+ 49us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 16:42:33.094171 (+ 13137us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36931/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 16:42:33.094470 (+ 299us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.12.126:33101: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36931/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":117,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.21.12.126:33101: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36931/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 16:42:33.096680 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 26672
2026-09-26T16:42:33Z chronyd exiting
I20260926 16:42:33.133834 21553 test_util.cc:175] -----------------------------------------------
I20260926 16:42:33.133909 21553 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440924937351-21553-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (304 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 16:42:33.134912 21553 mini_oidc.cc:152] Starting JWKS server
I20260926 16:42:33.135756 21553 webserver.cc:460] Webserver started at https://127.0.0.1:37315/ using document root <none> and password file <none>
I20260926 16:42:33.135870 21553 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 16:42:33.136139 21553 webserver.cc:460] Webserver started at http://127.0.0.1:43055/ using document root <none> and password file <none>
2026-09-26T16:42:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:42:33Z Disabled control of system clock
I20260926 16:42:33.144402 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:33735
--webserver_interface=127.21.12.126
--webserver_port=0
--builtin_ntp_servers=127.21.12.84:34921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:33735
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37315/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440924937351-21553-0/testchainca.cert with env {}
W20260926 16:42:33.248821 26775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:33.249145 26775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:33.249219 26775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:33.253227 26775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:42:33.253320 26775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:33.253348 26775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:33.253387 26775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:42:33.253417 26775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 16:42:33.253453 26775 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 16:42:33.253479 26775 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37315/jwks.json
I20260926 16:42:33.258641 26775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:34921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:33735
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:33735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37315/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440924937351-21553-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:33.259608 26775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:33.260555 26775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:33.268975 26781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:33.269049 26780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:33.269052 26783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:33.269219 26775 server_base.cc:1034] running on GCE node
I20260926 16:42:33.269834 26775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:33.270836 26775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:33.272001 26775 hybrid_clock.cc:648] HybridClock initialized: now 1790440953271986 us; error 33 us; skew 500 ppm
I20260926 16:42:33.273851 26775 webserver.cc:460] Webserver started at http://127.21.12.126:40541/ using document root <none> and password file <none>
I20260926 16:42:33.274399 26775 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:33.274461 26775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:33.274722 26775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:33.276652 26775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "6b7a662d114a42729718e7c8527a97b2"
format_stamp: "Formatted at 2026-09-26 16:42:33 on dist-test-slave-n1hw"
server_key: "1a0157182cd4f9f3deb99edbe85eee6c"
server_key_iv: "16b0ff4c2575f1fd132e06bd827d943f"
server_key_version: "encryptionkey@0"
I20260926 16:42:33.277209 26775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "6b7a662d114a42729718e7c8527a97b2"
format_stamp: "Formatted at 2026-09-26 16:42:33 on dist-test-slave-n1hw"
server_key: "1a0157182cd4f9f3deb99edbe85eee6c"
server_key_iv: "16b0ff4c2575f1fd132e06bd827d943f"
server_key_version: "encryptionkey@0"
I20260926 16:42:33.280647 26775 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 16:42:33.282955 26789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:33.284116 26775 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 16:42:33.284233 26775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "6b7a662d114a42729718e7c8527a97b2"
format_stamp: "Formatted at 2026-09-26 16:42:33 on dist-test-slave-n1hw"
server_key: "1a0157182cd4f9f3deb99edbe85eee6c"
server_key_iv: "16b0ff4c2575f1fd132e06bd827d943f"
server_key_version: "encryptionkey@0"
I20260926 16:42:33.284332 26775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:33.299633 26775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:33.300215 26775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:33.300400 26775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:33.307332 26775 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.126:33735
I20260926 16:42:33.307336 26841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.126:33735 every 8 connection(s)
I20260926 16:42:33.308264 26775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:42:33.309032 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 26775
I20260926 16:42:33.309161 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:42:33.309430 21553 external_mini_cluster.cc:1428] Setting key 302b7d3206fed3d9f493b4f1c274c446
I20260926 16:42:33.311691 26842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:33.317005 26842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b7a662d114a42729718e7c8527a97b2: Bootstrap starting.
I20260926 16:42:33.318244 26845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:33.310723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55916 (local address 127.21.12.126:33735)
0926 16:42:33.311335 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:33.311349 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:33.311382 (+ 33us) server_negotiation.cc:407] Connection header received
0926 16:42:33.312068 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:33.312085 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:33.312460 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:33.312806 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:33.313220 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:33.314223 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:33.315089 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:33.315551 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:33.315712 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:33.316590 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:33.316595 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:33.316607 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:33.316977 (+ 370us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:33.317034 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:33.317054 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:33.317162 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:33.317275 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 16:42:33.317519 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":136,"threads_started":1}
I20260926 16:42:33.319414 26842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b7a662d114a42729718e7c8527a97b2: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:33.320221 26842 log.cc:824] T 00000000000000000000000000000000 P 6b7a662d114a42729718e7c8527a97b2: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:33.321082 21553 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:42:33.322538 26842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b7a662d114a42729718e7c8527a97b2: No bootstrap required, opened a new log
I20260926 16:42:33.325901 26842 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b7a662d114a42729718e7c8527a97b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7a662d114a42729718e7c8527a97b2" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 33735 } }
I20260926 16:42:33.326133 26842 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b7a662d114a42729718e7c8527a97b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:33.326210 26842 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b7a662d114a42729718e7c8527a97b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b7a662d114a42729718e7c8527a97b2, State: Initialized, Role: FOLLOWER
I20260926 16:42:33.326659 26842 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b7a662d114a42729718e7c8527a97b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7a662d114a42729718e7c8527a97b2" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 33735 } }
I20260926 16:42:33.326805 26842 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b7a662d114a42729718e7c8527a97b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:42:33.326889 26842 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b7a662d114a42729718e7c8527a97b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:42:33.326982 26842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b7a662d114a42729718e7c8527a97b2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:33.327833 26842 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b7a662d114a42729718e7c8527a97b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7a662d114a42729718e7c8527a97b2" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 33735 } }
I20260926 16:42:33.328158 26842 leader_election.cc:302] T 00000000000000000000000000000000 P 6b7a662d114a42729718e7c8527a97b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b7a662d114a42729718e7c8527a97b2; no voters:
I20260926 16:42:33.328476 26842 leader_election.cc:288] T 00000000000000000000000000000000 P 6b7a662d114a42729718e7c8527a97b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:42:33.328545 26847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b7a662d114a42729718e7c8527a97b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:33.328730 26847 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b7a662d114a42729718e7c8527a97b2 [term 1 LEADER]: Becoming Leader. State: Replica: 6b7a662d114a42729718e7c8527a97b2, State: Running, Role: LEADER
I20260926 16:42:33.329023 26847 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b7a662d114a42729718e7c8527a97b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7a662d114a42729718e7c8527a97b2" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 33735 } }
I20260926 16:42:33.329720 26842 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b7a662d114a42729718e7c8527a97b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:42:33.330737 26848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b7a662d114a42729718e7c8527a97b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b7a662d114a42729718e7c8527a97b2. Latest consensus state: current_term: 1 leader_uuid: "6b7a662d114a42729718e7c8527a97b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7a662d114a42729718e7c8527a97b2" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 33735 } } }
I20260926 16:42:33.330767 26849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b7a662d114a42729718e7c8527a97b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b7a662d114a42729718e7c8527a97b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7a662d114a42729718e7c8527a97b2" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 33735 } } }
I20260926 16:42:33.330858 26848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b7a662d114a42729718e7c8527a97b2 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:33.330864 26849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b7a662d114a42729718e7c8527a97b2 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:33.331992 26856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:42:33.334556 26856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:42:33.339715 26856 catalog_manager.cc:1347] Generated new cluster ID: e099bb6dcd5646dab769f8da62978aa1
I20260926 16:42:33.339778 26856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:42:33.351058 26856 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:42:33.351913 26856 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:42:33.360997 26856 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b7a662d114a42729718e7c8527a97b2: Generated new TSK 0
I20260926 16:42:33.361634 26856 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 16:42:33.396232 26872 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 16:42:33.396324 26872 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 16:42:33.396947 26845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:33.381328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55918 (local address 127.21.12.126:33735)
0926 16:42:33.381459 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:33.381463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:33.381609 (+ 146us) server_negotiation.cc:407] Connection header received
0926 16:42:33.381733 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:33.381736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:33.381810 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:33.381928 (+ 118us) server_negotiation.cc:226] Negotiated authn=JWT
0926 16:42:33.382144 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:33.382958 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:33.383774 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:33.384117 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:33.384216 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:33.384468 (+ 252us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 16:42:33.396581 (+ 12113us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37315/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 16:42:33.396806 (+ 225us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55918: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37315/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":50}
W20260926 16:42:33.397009 26845 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55918: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37315/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260926 16:42:33.397038 26871 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:33.381209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:33735 (local address 127.0.0.1:55918)
0926 16:42:33.381528 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:33.381540 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:33.381633 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:33.381930 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:33.381933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:33.381939 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0926 16:42:33.382040 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:33.382045 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:33.383071 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:33.383073 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:33.383669 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:33.383676 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:33.384228 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:33.384231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:33.384290 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:33.384311 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 16:42:33.396753 (+ 12442us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37315/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 16:42:33.396872 (+ 119us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.12.126:33735: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37315/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":102,"threads_started":1}
I20260926 16:42:33.397769 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 26775
2026-09-26T16:42:33Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (301 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 16:42:33.436904 21553 mini_oidc.cc:152] Starting JWKS server
I20260926 16:42:33.437825 21553 webserver.cc:460] Webserver started at https://127.0.0.1:36977/ using document root <none> and password file <none>
I20260926 16:42:33.437968 21553 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 16:42:33.438237 21553 webserver.cc:460] Webserver started at http://127.0.0.1:38529/ using document root <none> and password file <none>
2026-09-26T16:42:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:42:33Z Disabled control of system clock
I20260926 16:42:33.447225 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:38771
--webserver_interface=127.21.12.126
--webserver_port=0
--builtin_ntp_servers=127.21.12.84:39327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:38771
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36977/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 16:42:33.553668 26878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:33.553982 26878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:33.554020 26878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:33.557819 26878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:42:33.557865 26878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:33.557886 26878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:33.557902 26878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:42:33.557917 26878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 16:42:33.557932 26878 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 16:42:33.557950 26878 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36977/jwks.json
I20260926 16:42:33.562574 26878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:39327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:38771
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:38771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36977/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:33.563493 26878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:33.564740 26878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:33.572939 26884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:33.572992 26883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:33.573056 26886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:33.573077 26878 server_base.cc:1034] running on GCE node
I20260926 16:42:33.573939 26878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:33.575026 26878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:33.576218 26878 hybrid_clock.cc:648] HybridClock initialized: now 1790440953576209 us; error 29 us; skew 500 ppm
I20260926 16:42:33.578142 26878 webserver.cc:460] Webserver started at http://127.21.12.126:40235/ using document root <none> and password file <none>
I20260926 16:42:33.578716 26878 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:33.578807 26878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:33.579042 26878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:33.580976 26878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "c99f4ac4ddfc49269a0e66fbbec14b4e"
format_stamp: "Formatted at 2026-09-26 16:42:33 on dist-test-slave-n1hw"
server_key: "9ab1200a62d9bedc6bca8c2dcc11543a"
server_key_iv: "7f352c51a93eb65de671114970758a18"
server_key_version: "encryptionkey@0"
I20260926 16:42:33.581528 26878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "c99f4ac4ddfc49269a0e66fbbec14b4e"
format_stamp: "Formatted at 2026-09-26 16:42:33 on dist-test-slave-n1hw"
server_key: "9ab1200a62d9bedc6bca8c2dcc11543a"
server_key_iv: "7f352c51a93eb65de671114970758a18"
server_key_version: "encryptionkey@0"
I20260926 16:42:33.585006 26878 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 16:42:33.587353 26892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:33.588613 26878 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 16:42:33.588760 26878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "c99f4ac4ddfc49269a0e66fbbec14b4e"
format_stamp: "Formatted at 2026-09-26 16:42:33 on dist-test-slave-n1hw"
server_key: "9ab1200a62d9bedc6bca8c2dcc11543a"
server_key_iv: "7f352c51a93eb65de671114970758a18"
server_key_version: "encryptionkey@0"
I20260926 16:42:33.588874 26878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:33.600878 26878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:33.601469 26878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:33.601644 26878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:33.608070 26878 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.126:38771
I20260926 16:42:33.608103 26944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.126:38771 every 8 connection(s)
I20260926 16:42:33.608927 26878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:42:33.611860 26945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:33.612426 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 26878
I20260926 16:42:33.612555 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:42:33.612841 21553 external_mini_cluster.cc:1428] Setting key b09b0a2048f394f641e0a607e63b7e10
I20260926 16:42:33.617559 26945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c99f4ac4ddfc49269a0e66fbbec14b4e: Bootstrap starting.
I20260926 16:42:33.619630 26945 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c99f4ac4ddfc49269a0e66fbbec14b4e: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:33.620306 26945 log.cc:824] T 00000000000000000000000000000000 P c99f4ac4ddfc49269a0e66fbbec14b4e: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:33.621543 26948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:33.614152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37832 (local address 127.21.12.126:38771)
0926 16:42:33.614759 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:33.614771 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:33.614804 (+ 33us) server_negotiation.cc:407] Connection header received
0926 16:42:33.615432 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:33.615448 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:33.615814 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:33.616148 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:33.616541 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:33.617457 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:33.618442 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:33.618890 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:33.619081 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:33.619928 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:33.619933 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:33.619945 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:33.620305 (+ 360us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:33.620357 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:33.620403 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:33.620502 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:33.620653 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 16:42:33.620885 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":114,"threads_started":1}
I20260926 16:42:33.622171 26945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c99f4ac4ddfc49269a0e66fbbec14b4e: No bootstrap required, opened a new log
I20260926 16:42:33.624325 21553 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:42:33.626343 26945 raft_consensus.cc:348] T 00000000000000000000000000000000 P c99f4ac4ddfc49269a0e66fbbec14b4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99f4ac4ddfc49269a0e66fbbec14b4e" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 38771 } }
I20260926 16:42:33.626572 26945 raft_consensus.cc:374] T 00000000000000000000000000000000 P c99f4ac4ddfc49269a0e66fbbec14b4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:33.626653 26945 raft_consensus.cc:729] T 00000000000000000000000000000000 P c99f4ac4ddfc49269a0e66fbbec14b4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c99f4ac4ddfc49269a0e66fbbec14b4e, State: Initialized, Role: FOLLOWER
I20260926 16:42:33.627027 26945 consensus_queue.cc:260] T 00000000000000000000000000000000 P c99f4ac4ddfc49269a0e66fbbec14b4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99f4ac4ddfc49269a0e66fbbec14b4e" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 38771 } }
I20260926 16:42:33.627166 26945 raft_consensus.cc:388] T 00000000000000000000000000000000 P c99f4ac4ddfc49269a0e66fbbec14b4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:42:33.627249 26945 raft_consensus.cc:482] T 00000000000000000000000000000000 P c99f4ac4ddfc49269a0e66fbbec14b4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:42:33.627341 26945 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c99f4ac4ddfc49269a0e66fbbec14b4e [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:33.628144 26945 raft_consensus.cc:504] T 00000000000000000000000000000000 P c99f4ac4ddfc49269a0e66fbbec14b4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99f4ac4ddfc49269a0e66fbbec14b4e" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 38771 } }
I20260926 16:42:33.628466 26945 leader_election.cc:302] T 00000000000000000000000000000000 P c99f4ac4ddfc49269a0e66fbbec14b4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c99f4ac4ddfc49269a0e66fbbec14b4e; no voters:
I20260926 16:42:33.628746 26945 leader_election.cc:288] T 00000000000000000000000000000000 P c99f4ac4ddfc49269a0e66fbbec14b4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:42:33.628887 26950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c99f4ac4ddfc49269a0e66fbbec14b4e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:33.629122 26950 raft_consensus.cc:686] T 00000000000000000000000000000000 P c99f4ac4ddfc49269a0e66fbbec14b4e [term 1 LEADER]: Becoming Leader. State: Replica: c99f4ac4ddfc49269a0e66fbbec14b4e, State: Running, Role: LEADER
I20260926 16:42:33.629477 26950 consensus_queue.cc:237] T 00000000000000000000000000000000 P c99f4ac4ddfc49269a0e66fbbec14b4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99f4ac4ddfc49269a0e66fbbec14b4e" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 38771 } }
I20260926 16:42:33.630008 26945 sys_catalog.cc:564] T 00000000000000000000000000000000 P c99f4ac4ddfc49269a0e66fbbec14b4e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:42:33.631173 26951 sys_catalog.cc:455] T 00000000000000000000000000000000 P c99f4ac4ddfc49269a0e66fbbec14b4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c99f4ac4ddfc49269a0e66fbbec14b4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99f4ac4ddfc49269a0e66fbbec14b4e" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 38771 } } }
I20260926 16:42:33.631174 26952 sys_catalog.cc:455] T 00000000000000000000000000000000 P c99f4ac4ddfc49269a0e66fbbec14b4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c99f4ac4ddfc49269a0e66fbbec14b4e. Latest consensus state: current_term: 1 leader_uuid: "c99f4ac4ddfc49269a0e66fbbec14b4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99f4ac4ddfc49269a0e66fbbec14b4e" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 38771 } } }
I20260926 16:42:33.631332 26951 sys_catalog.cc:458] T 00000000000000000000000000000000 P c99f4ac4ddfc49269a0e66fbbec14b4e [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:33.631332 26952 sys_catalog.cc:458] T 00000000000000000000000000000000 P c99f4ac4ddfc49269a0e66fbbec14b4e [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:33.631901 26959 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:42:33.634466 26959 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:42:33.639097 26959 catalog_manager.cc:1347] Generated new cluster ID: b39f6a82f9df490a82c26df2b6e4c8bb
I20260926 16:42:33.639156 26959 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:42:33.648136 26959 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:42:33.649225 26959 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:42:33.660564 26959 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c99f4ac4ddfc49269a0e66fbbec14b4e: Generated new TSK 0
I20260926 16:42:33.661206 26959 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:42:33.676136 26948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:33.671347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37842 (local address 127.21.12.126:38771)
0926 16:42:33.671488 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:33.671495 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:33.671656 (+ 161us) server_negotiation.cc:407] Connection header received
0926 16:42:33.671810 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:33.671812 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:33.671857 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:33.671946 (+ 89us) server_negotiation.cc:226] Negotiated authn=JWT
0926 16:42:33.672206 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:33.672763 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:33.674019 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:33.674229 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:33.674308 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:33.674629 (+ 321us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 16:42:33.675739 (+ 1110us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36977/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 16:42:33.675975 (+ 236us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37842: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36977/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 16:42:33.676215 26974 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:33.671263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38771 (local address 127.0.0.1:37842)
0926 16:42:33.671565 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:33.671580 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:33.671693 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:33.671981 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:33.671985 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:33.671992 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0926 16:42:33.672106 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:33.672112 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:33.672881 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:33.672884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:33.673899 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:33.673910 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:33.674351 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:33.674355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:33.674445 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:33.674478 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 16:42:33.675910 (+ 1432us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36977/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 16:42:33.676048 (+ 138us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.12.126:38771: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36977/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":107,"threads_started":1}
W20260926 16:42:33.676569 26948 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37842: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36977/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260926 16:42:33.676961 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 26878
E20260926 16:42:33.688804 26975 webserver.cc:533] Webserver: sslize failed:
E20260926 16:42:33.688877 26975 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-26T16:42:33Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (279 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T16:42:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:42:33Z Disabled control of system clock
I20260926 16:42:33.725014 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:46817
--webserver_interface=127.21.12.126
--webserver_port=0
--builtin_ntp_servers=127.21.12.84:39903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:46817
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440924937351-21553-0/testchainca.cert with env {}
W20260926 16:42:33.832293 26979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:33.832574 26979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:33.832652 26979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:33.836570 26979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:42:33.836648 26979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:33.836678 26979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:33.836716 26979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:42:33.836748 26979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 16:42:33.836784 26979 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 16:42:33.836810 26979 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 16:42:33.842036 26979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:39903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:46817
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:46817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440924937351-21553-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:33.843030 26979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:33.844005 26979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:33.852571 26985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:33.852556 26987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:33.852560 26984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:33.852610 26979 server_base.cc:1034] running on GCE node
I20260926 16:42:33.853410 26979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:33.854460 26979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:33.855625 26979 hybrid_clock.cc:648] HybridClock initialized: now 1790440953855612 us; error 23 us; skew 500 ppm
I20260926 16:42:33.861447 26979 webserver.cc:460] Webserver started at http://127.21.12.126:33629/ using document root <none> and password file <none>
I20260926 16:42:33.862178 26979 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:33.862339 26979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:33.862602 26979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:33.864864 26979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "cfd3a24aec244b209a924746d86faca1"
format_stamp: "Formatted at 2026-09-26 16:42:33 on dist-test-slave-n1hw"
server_key: "e57801c5e366d3f2e3fe5289614dfd92"
server_key_iv: "ec509c747bdbd4ef8894bae1a8c4fe9f"
server_key_version: "encryptionkey@0"
I20260926 16:42:33.865509 26979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "cfd3a24aec244b209a924746d86faca1"
format_stamp: "Formatted at 2026-09-26 16:42:33 on dist-test-slave-n1hw"
server_key: "e57801c5e366d3f2e3fe5289614dfd92"
server_key_iv: "ec509c747bdbd4ef8894bae1a8c4fe9f"
server_key_version: "encryptionkey@0"
I20260926 16:42:33.869688 26979 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:42:33.872295 26993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:33.873584 26979 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 16:42:33.873741 26979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "cfd3a24aec244b209a924746d86faca1"
format_stamp: "Formatted at 2026-09-26 16:42:33 on dist-test-slave-n1hw"
server_key: "e57801c5e366d3f2e3fe5289614dfd92"
server_key_iv: "ec509c747bdbd4ef8894bae1a8c4fe9f"
server_key_version: "encryptionkey@0"
I20260926 16:42:33.873925 26979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:33.895327 26979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:33.896283 26979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:33.896516 26979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:33.904703 27045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.126:46817 every 8 connection(s)
I20260926 16:42:33.904701 26979 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.126:46817
I20260926 16:42:33.905825 26979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:42:33.908851 27046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:33.910547 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 26979
I20260926 16:42:33.910705 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:42:33.911036 21553 external_mini_cluster.cc:1428] Setting key cf522befc94cf9d8c9d478a34b67d7b8
I20260926 16:42:33.915334 27046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfd3a24aec244b209a924746d86faca1: Bootstrap starting.
I20260926 16:42:33.917857 27046 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfd3a24aec244b209a924746d86faca1: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:33.918708 27046 log.cc:824] T 00000000000000000000000000000000 P cfd3a24aec244b209a924746d86faca1: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:33.920976 27046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfd3a24aec244b209a924746d86faca1: No bootstrap required, opened a new log
I20260926 16:42:33.921716 27049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:33.912518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36264 (local address 127.21.12.126:46817)
0926 16:42:33.913184 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:33.913205 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:33.913246 (+ 41us) server_negotiation.cc:407] Connection header received
0926 16:42:33.914263 (+ 1017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:33.914286 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:33.914779 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:33.915132 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:33.915569 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:33.917010 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:33.918130 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:33.918817 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:33.918968 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:33.919789 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:33.919795 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:33.919810 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:33.920268 (+ 458us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:33.920333 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:33.920359 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:33.920468 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:33.920569 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 16:42:33.920821 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":106,"threads_started":1}
I20260926 16:42:33.924609 21553 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:42:33.925067 27046 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfd3a24aec244b209a924746d86faca1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfd3a24aec244b209a924746d86faca1" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46817 } }
I20260926 16:42:33.925262 27046 raft_consensus.cc:374] T 00000000000000000000000000000000 P cfd3a24aec244b209a924746d86faca1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:33.925318 27046 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfd3a24aec244b209a924746d86faca1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfd3a24aec244b209a924746d86faca1, State: Initialized, Role: FOLLOWER
I20260926 16:42:33.925727 27046 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfd3a24aec244b209a924746d86faca1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfd3a24aec244b209a924746d86faca1" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46817 } }
I20260926 16:42:33.925889 27046 raft_consensus.cc:388] T 00000000000000000000000000000000 P cfd3a24aec244b209a924746d86faca1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:42:33.925953 27046 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfd3a24aec244b209a924746d86faca1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:42:33.926056 27046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfd3a24aec244b209a924746d86faca1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:33.926856 27046 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfd3a24aec244b209a924746d86faca1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfd3a24aec244b209a924746d86faca1" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46817 } }
I20260926 16:42:33.927132 27046 leader_election.cc:302] T 00000000000000000000000000000000 P cfd3a24aec244b209a924746d86faca1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cfd3a24aec244b209a924746d86faca1; no voters:
I20260926 16:42:33.927366 27046 leader_election.cc:288] T 00000000000000000000000000000000 P cfd3a24aec244b209a924746d86faca1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:42:33.927484 27051 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfd3a24aec244b209a924746d86faca1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:33.927753 27051 raft_consensus.cc:686] T 00000000000000000000000000000000 P cfd3a24aec244b209a924746d86faca1 [term 1 LEADER]: Becoming Leader. State: Replica: cfd3a24aec244b209a924746d86faca1, State: Running, Role: LEADER
I20260926 16:42:33.928131 27051 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfd3a24aec244b209a924746d86faca1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfd3a24aec244b209a924746d86faca1" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46817 } }
I20260926 16:42:33.928586 27046 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfd3a24aec244b209a924746d86faca1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:42:33.929782 27053 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfd3a24aec244b209a924746d86faca1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfd3a24aec244b209a924746d86faca1. Latest consensus state: current_term: 1 leader_uuid: "cfd3a24aec244b209a924746d86faca1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfd3a24aec244b209a924746d86faca1" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46817 } } }
I20260926 16:42:33.929903 27053 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfd3a24aec244b209a924746d86faca1 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:33.930310 27060 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:42:33.930281 27052 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfd3a24aec244b209a924746d86faca1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cfd3a24aec244b209a924746d86faca1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfd3a24aec244b209a924746d86faca1" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46817 } } }
I20260926 16:42:33.930496 27052 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfd3a24aec244b209a924746d86faca1 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:33.933472 27060 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:42:33.938915 27060 catalog_manager.cc:1347] Generated new cluster ID: 7293d496e2bc4e3a93e8e65e2f237aba
I20260926 16:42:33.938980 27060 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:42:33.949007 27060 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:42:33.949895 27060 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:42:33.956425 27060 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cfd3a24aec244b209a924746d86faca1: Generated new TSK 0
I20260926 16:42:33.957085 27060 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:42:34.049216 27049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:33.966379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36276 (local address 127.21.12.126:46817)
0926 16:42:33.966520 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:33.966524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:33.966730 (+ 206us) server_negotiation.cc:407] Connection header received
0926 16:42:33.966897 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:33.966900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:33.966945 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:33.967029 (+ 84us) server_negotiation.cc:226] Negotiated authn=JWT
0926 16:42:33.967290 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:33.967840 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:33.969064 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:33.969283 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:33.969368 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:33.969701 (+ 333us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 16:42:34.048710 (+ 79009us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 16:42:34.049050 (+ 340us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36276: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 16:42:34.049302 27049 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36276: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 16:42:34.049366 27075 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:33.966316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:46817 (local address 127.0.0.1:36276)
0926 16:42:33.966633 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:33.966649 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:33.966763 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:33.967060 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:33.967063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:33.967072 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0926 16:42:33.967188 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:33.967194 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:33.967957 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:33.967961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:33.968931 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:33.968942 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:33.969411 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:33.969415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:33.969523 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:33.969551 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 16:42:34.048950 (+ 79399us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 16:42:34.049171 (+ 221us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.12.126:46817: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":117,"threads_started":1}
I20260926 16:42:34.050359 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 26979
2026-09-26T16:42:34Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (352 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (120 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (120 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:42:34 dist-test-slave-n1hw krb5kdc[27087](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:42:34 dist-test-slave-n1hw krb5kdc[27087](info): set up 2 sockets
Sep 26 16:42:34 dist-test-slave-n1hw krb5kdc[27087](info): commencing operation
krb5kdc: starting...
W20260926 16:42:35.336216 21553 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Sep 26 16:42:35 dist-test-slave-n1hw krb5kdc[27087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:42:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:42:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
I20260926 16:42:35.460328 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:45787
--webserver_interface=127.21.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:44671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:45787
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:35.568645 27100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:35.568899 27100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:35.568938 27100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:35.573678 27100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:42:35.573779 27100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:35.573812 27100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:35.573880 27100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:42:35.573920 27100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:42:35.579794 27100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:44671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:45787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:45787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:35.580940 27100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:35.582053 27100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:35.590497 27106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:35.590505 27105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:35.590633 27100 server_base.cc:1034] running on GCE node
W20260926 16:42:35.590520 27108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:35.591468 27100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:35.592470 27100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:35.593667 27100 hybrid_clock.cc:648] HybridClock initialized: now 1790440955593626 us; error 58 us; skew 500 ppm
Sep 26 16:42:35 dist-test-slave-n1hw krb5kdc[27087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440955, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:35.596374 27100 init.cc:377] Logged in from keytab as kudu/127.21.12.126@KRBTEST.COM (short username kudu)
I20260926 16:42:35.597208 27100 webserver.cc:460] Webserver started at http://127.21.12.126:43293/ using document root <none> and password file <none>
I20260926 16:42:35.597836 27100 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:35.597927 27100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:35.598151 27100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:35.600605 27100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "fd834598e17b487b9d744846f841731b"
format_stamp: "Formatted at 2026-09-26 16:42:35 on dist-test-slave-n1hw"
server_key: "e93ac5193fd0a03463fbf6b35ef67cb9"
server_key_iv: "9573e7c8eaa056b3735beda2ce10cd48"
server_key_version: "encryptionkey@0"
I20260926 16:42:35.601207 27100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "fd834598e17b487b9d744846f841731b"
format_stamp: "Formatted at 2026-09-26 16:42:35 on dist-test-slave-n1hw"
server_key: "e93ac5193fd0a03463fbf6b35ef67cb9"
server_key_iv: "9573e7c8eaa056b3735beda2ce10cd48"
server_key_version: "encryptionkey@0"
I20260926 16:42:35.604807 27100 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 16:42:35.607209 27115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:35.608410 27100 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260926 16:42:35.608561 27100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "fd834598e17b487b9d744846f841731b"
format_stamp: "Formatted at 2026-09-26 16:42:35 on dist-test-slave-n1hw"
server_key: "e93ac5193fd0a03463fbf6b35ef67cb9"
server_key_iv: "9573e7c8eaa056b3735beda2ce10cd48"
server_key_version: "encryptionkey@0"
I20260926 16:42:35.608676 27100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:35.627720 27100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:35.630662 27100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:35.630857 27100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:35.637640 27100 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.126:45787
I20260926 16:42:35.637689 27167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.126:45787 every 8 connection(s)
I20260926 16:42:35.638568 27100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:42:35.641166 27168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:35.646333 27168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd834598e17b487b9d744846f841731b: Bootstrap starting.
I20260926 16:42:35.646685 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 27100
I20260926 16:42:35.646816 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:42:35.647104 21553 external_mini_cluster.cc:1428] Setting key c310ef3315fa8a1e49d1dc9974dc5693
I20260926 16:42:35.648839 27168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd834598e17b487b9d744846f841731b: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:35.649559 27168 log.cc:824] T 00000000000000000000000000000000 P fd834598e17b487b9d744846f841731b: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:35.651331 27168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd834598e17b487b9d744846f841731b: No bootstrap required, opened a new log
I20260926 16:42:35.654803 27168 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd834598e17b487b9d744846f841731b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd834598e17b487b9d744846f841731b" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 45787 } }
I20260926 16:42:35.655063 27168 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd834598e17b487b9d744846f841731b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:35.655131 27168 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd834598e17b487b9d744846f841731b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd834598e17b487b9d744846f841731b, State: Initialized, Role: FOLLOWER
Sep 26 16:42:35 dist-test-slave-n1hw krb5kdc[27087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:42:35.655589 27168 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd834598e17b487b9d744846f841731b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd834598e17b487b9d744846f841731b" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 45787 } }
I20260926 16:42:35.655769 27168 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd834598e17b487b9d744846f841731b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:42:35.655845 27168 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd834598e17b487b9d744846f841731b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:42:35.655949 27168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd834598e17b487b9d744846f841731b [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:35.656960 27168 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd834598e17b487b9d744846f841731b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd834598e17b487b9d744846f841731b" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 45787 } }
I20260926 16:42:35.657322 27168 leader_election.cc:302] T 00000000000000000000000000000000 P fd834598e17b487b9d744846f841731b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd834598e17b487b9d744846f841731b; no voters:
I20260926 16:42:35.657618 27168 leader_election.cc:288] T 00000000000000000000000000000000 P fd834598e17b487b9d744846f841731b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:42:35.657716 27173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd834598e17b487b9d744846f841731b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:35.657944 27173 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd834598e17b487b9d744846f841731b [term 1 LEADER]: Becoming Leader. State: Replica: fd834598e17b487b9d744846f841731b, State: Running, Role: LEADER
I20260926 16:42:35.658252 27173 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd834598e17b487b9d744846f841731b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd834598e17b487b9d744846f841731b" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 45787 } }
I20260926 16:42:35.658888 27168 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd834598e17b487b9d744846f841731b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:42:35.659921 27175 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd834598e17b487b9d744846f841731b [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd834598e17b487b9d744846f841731b. Latest consensus state: current_term: 1 leader_uuid: "fd834598e17b487b9d744846f841731b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd834598e17b487b9d744846f841731b" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 45787 } } }
I20260926 16:42:35.660038 27175 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd834598e17b487b9d744846f841731b [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:35.660312 27174 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd834598e17b487b9d744846f841731b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd834598e17b487b9d744846f841731b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd834598e17b487b9d744846f841731b" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 45787 } } }
I20260926 16:42:35.660405 27174 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd834598e17b487b9d744846f841731b [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:35.660884 27182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:42:35.661697 27171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:35.648350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35814 (local address 127.21.12.126:45787)
0926 16:42:35.648736 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:35.648750 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:35.648783 (+ 33us) server_negotiation.cc:407] Connection header received
0926 16:42:35.649462 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:35.649480 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:35.649889 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:35.650197 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:35.650966 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:35.651951 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:35.652796 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:35.653244 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:35.653440 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:35.656191 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:35.656196 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:35.656206 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:35.656230 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:35.658215 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:35.658726 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:35.658731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:35.658737 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:35.658875 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:35.659218 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:35.659221 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:35.659224 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:35.659723 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:35.659933 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:35.660388 (+ 455us) server_negotiation.cc:299] Negotiation successful
0926 16:42:35.660676 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":118,"threads_started":1}
I20260926 16:42:35.664366 27182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:42:35.669689 27182 catalog_manager.cc:1347] Generated new cluster ID: da28285d0e2242e0888588e803975cac
I20260926 16:42:35.669790 27182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:42:35.678750 27182 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:42:35.679641 27182 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:42:35.684604 27182 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd834598e17b487b9d744846f841731b: Generated new TSK 0
I20260926 16:42:35.685189 27182 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab.
Entry for principal kudu/127.21.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab.
I20260926 16:42:35.716174 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.65:0
--local_ip_for_outbound_sockets=127.21.12.65
--webserver_interface=127.21.12.65
--webserver_port=0
--tserver_master_addrs=127.21.12.126:45787
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab
--principal=kudu/127.21.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:44671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:35.836931 27194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:35.837157 27194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:35.837198 27194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:35.841123 27194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:35.841173 27194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:35.841229 27194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.65
I20260926 16:42:35.846299 27194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:44671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab
--principal=kudu/127.21.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.65
--webserver_port=0
--tserver_master_addrs=127.21.12.126:45787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.65
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:35.847278 27194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:35.848241 27194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:35.857614 27202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:35.857640 27200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:35.857699 27194 server_base.cc:1034] running on GCE node
W20260926 16:42:35.857875 27199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:35.858263 27194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:35.858889 27194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:35.860073 27194 hybrid_clock.cc:648] HybridClock initialized: now 1790440955860036 us; error 47 us; skew 500 ppm
Sep 26 16:42:35 dist-test-slave-n1hw krb5kdc[27087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440955, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:35.862860 27194 init.cc:377] Logged in from keytab as kudu/127.21.12.65@KRBTEST.COM (short username kudu)
I20260926 16:42:35.863698 27194 webserver.cc:460] Webserver started at http://127.21.12.65:36797/ using document root <none> and password file <none>
I20260926 16:42:35.864229 27194 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:35.864316 27194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:35.864534 27194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:35.866503 27194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-0/data/instance:
uuid: "c3a6ded0d37a4affb400a18cc3f7bb08"
format_stamp: "Formatted at 2026-09-26 16:42:35 on dist-test-slave-n1hw"
server_key: "5027458073a9b5c2617f3b010892375d"
server_key_iv: "6cc783d1044bccdbb04f709cb2336130"
server_key_version: "encryptionkey@0"
I20260926 16:42:35.867033 27194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance:
uuid: "c3a6ded0d37a4affb400a18cc3f7bb08"
format_stamp: "Formatted at 2026-09-26 16:42:35 on dist-test-slave-n1hw"
server_key: "5027458073a9b5c2617f3b010892375d"
server_key_iv: "6cc783d1044bccdbb04f709cb2336130"
server_key_version: "encryptionkey@0"
I20260926 16:42:35.870585 27194 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:42:35.873008 27209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:35.874408 27194 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:42:35.874537 27194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-0/wal
uuid: "c3a6ded0d37a4affb400a18cc3f7bb08"
format_stamp: "Formatted at 2026-09-26 16:42:35 on dist-test-slave-n1hw"
server_key: "5027458073a9b5c2617f3b010892375d"
server_key_iv: "6cc783d1044bccdbb04f709cb2336130"
server_key_version: "encryptionkey@0"
I20260926 16:42:35.874642 27194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:35.888434 27194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:35.891554 27194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:35.891747 27194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:35.892303 27194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:42:35.893086 27194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:35.893157 27194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:35.893225 27194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:35.893270 27194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:35.902173 27194 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.65:34555
I20260926 16:42:35.902225 27322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.65:34555 every 8 connection(s)
I20260926 16:42:35.903029 27194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
Sep 26 16:42:35 dist-test-slave-n1hw krb5kdc[27087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440955, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:42:35.912451 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 27194
I20260926 16:42:35.912591 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance
I20260926 16:42:35.912927 21553 external_mini_cluster.cc:1428] Setting key 7a0d6faa59839fe84b55112b22b81d77
I20260926 16:42:35.917059 27171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:35.904839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:41473 (local address 127.21.12.126:45787)
0926 16:42:35.904985 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:35.904989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:35.905936 (+ 947us) server_negotiation.cc:407] Connection header received
0926 16:42:35.907200 (+ 1264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:35.907203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:35.907245 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:35.907317 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:35.908399 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:35.908912 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:35.910063 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:35.910298 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:35.910386 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:35.912931 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:35.912953 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:35.912955 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:35.912977 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:35.914876 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:35.915373 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:35.915380 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:35.915385 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:35.915481 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:35.915807 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:35.915813 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:35.915818 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:35.916063 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:35.916163 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:35.916852 (+ 689us) server_negotiation.cc:299] Negotiation successful
0926 16:42:35.916946 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 16:42:35.918215 27325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:35.905247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:45787 (local address 127.21.12.65:41473)
0926 16:42:35.905710 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:35.905818 (+ 108us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:35.907008 (+ 1190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:35.907481 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:35.907488 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:35.907813 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:35.908272 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:35.908285 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:35.909038 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:35.909041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:35.909941 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:35.909952 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:35.910421 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:35.910424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:35.910568 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:35.911154 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:35.911169 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:35.912766 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:35.915001 (+ 2235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:35.915005 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:35.915016 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:35.915266 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:35.915572 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:35.915574 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:35.915576 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:35.915715 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:35.916196 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:35.916204 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:35.916581 (+ 377us) client_negotiation.cc:769] Sending connection context
0926 16:42:35.916798 (+ 217us) client_negotiation.cc:240] Negotiation successful
0926 16:42:35.917139 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":115,"threads_started":1}
I20260926 16:42:35.919526 27323 heartbeater.cc:347] Connected to a master server at 127.21.12.126:45787
I20260926 16:42:35.919776 27323 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:35.920249 27323 heartbeater.cc:511] Master 127.21.12.126:45787 requested a full tablet report, sending...
I20260926 16:42:35.921702 27132 ts_manager.cc:194] Registered new tserver with Master: c3a6ded0d37a4affb400a18cc3f7bb08 (127.21.12.65:34555)
WARNING: no policy specified for kudu/127.21.12.66@KRBTEST.COM; defaulting to no policy
I20260926 16:42:35.922921 27132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.65@KRBTEST.COM'} at 127.21.12.65:41473
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab.
Entry for principal kudu/127.21.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab.
I20260926 16:42:35.940876 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.66:0
--local_ip_for_outbound_sockets=127.21.12.66
--webserver_interface=127.21.12.66
--webserver_port=0
--tserver_master_addrs=127.21.12.126:45787
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab
--principal=kudu/127.21.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:44671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:36.048049 27328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:36.048295 27328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:36.048337 27328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:36.052258 27328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:36.052320 27328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:36.052377 27328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.66
I20260926 16:42:36.057631 27328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:44671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab
--principal=kudu/127.21.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.66
--webserver_port=0
--tserver_master_addrs=127.21.12.126:45787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.66
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:36.058766 27328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:36.059810 27328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:36.069380 27334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:36.069389 27336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:36.069391 27333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:36.069774 27328 server_base.cc:1034] running on GCE node
I20260926 16:42:36.070140 27328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:36.070755 27328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:36.071930 27328 hybrid_clock.cc:648] HybridClock initialized: now 1790440956071882 us; error 52 us; skew 500 ppm
Sep 26 16:42:36 dist-test-slave-n1hw krb5kdc[27087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440956, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:36.074899 27328 init.cc:377] Logged in from keytab as kudu/127.21.12.66@KRBTEST.COM (short username kudu)
I20260926 16:42:36.075847 27328 webserver.cc:460] Webserver started at http://127.21.12.66:45255/ using document root <none> and password file <none>
I20260926 16:42:36.076411 27328 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:36.076499 27328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:36.076726 27328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:36.078868 27328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-1/data/instance:
uuid: "ecdc74d641cd47cc8056f74101335359"
format_stamp: "Formatted at 2026-09-26 16:42:36 on dist-test-slave-n1hw"
server_key: "680f719b51fdf5c78e24a3f1c4b16496"
server_key_iv: "e0a8b7c719047ae02595c2468111fc3c"
server_key_version: "encryptionkey@0"
I20260926 16:42:36.079422 27328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance:
uuid: "ecdc74d641cd47cc8056f74101335359"
format_stamp: "Formatted at 2026-09-26 16:42:36 on dist-test-slave-n1hw"
server_key: "680f719b51fdf5c78e24a3f1c4b16496"
server_key_iv: "e0a8b7c719047ae02595c2468111fc3c"
server_key_version: "encryptionkey@0"
I20260926 16:42:36.082995 27328 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:42:36.085249 27343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:36.086519 27328 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:42:36.086653 27328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-1/wal
uuid: "ecdc74d641cd47cc8056f74101335359"
format_stamp: "Formatted at 2026-09-26 16:42:36 on dist-test-slave-n1hw"
server_key: "680f719b51fdf5c78e24a3f1c4b16496"
server_key_iv: "e0a8b7c719047ae02595c2468111fc3c"
server_key_version: "encryptionkey@0"
I20260926 16:42:36.086758 27328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:36.099498 27328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:36.103050 27328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:36.103262 27328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:36.103849 27328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:42:36.104672 27328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:36.104748 27328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:36.104830 27328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:36.104888 27328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:36.114991 27328 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.66:42477
I20260926 16:42:36.115022 27456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.66:42477 every 8 connection(s)
I20260926 16:42:36.115891 27328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
I20260926 16:42:36.116729 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 27328
I20260926 16:42:36.116852 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance
I20260926 16:42:36.117175 21553 external_mini_cluster.cc:1428] Setting key 42255bb17bd7dfeda40e89dbee9b4ebc
Sep 26 16:42:36 dist-test-slave-n1hw krb5kdc[27087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440956, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.66@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.12.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.67@KRBTEST.COM" created.
I20260926 16:42:36.131902 27171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:36.117843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:35465 (local address 127.21.12.126:45787)
0926 16:42:36.117984 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:36.117989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:36.118961 (+ 972us) server_negotiation.cc:407] Connection header received
0926 16:42:36.120403 (+ 1442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:36.120409 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:36.120476 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:36.120570 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:36.122311 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:36.122885 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:36.124825 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:36.125052 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:36.125136 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:36.128020 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:36.128024 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:36.128025 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:36.128049 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:36.129828 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:36.130411 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:36.130415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:36.130417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:36.130499 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:36.130839 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:36.130849 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:36.130851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:36.131027 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:36.131134 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:36.131608 (+ 474us) server_negotiation.cc:299] Negotiation successful
0926 16:42:36.131724 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 16:42:36.132661 27459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:36.118270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:45787 (local address 127.21.12.66:35465)
0926 16:42:36.118809 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:36.118869 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:36.120116 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:36.120796 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:36.120806 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:36.121413 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:36.122132 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:36.122156 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:36.123072 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:36.123078 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:36.124665 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:36.124681 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:36.125214 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:36.125219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:36.125462 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:36.126140 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:36.126167 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:36.127858 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:36.129961 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:36.129967 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:36.129980 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:36.130296 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:36.130597 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:36.130600 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:36.130602 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:36.130737 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:36.131130 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:36.131136 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:36.131377 (+ 241us) client_negotiation.cc:769] Sending connection context
0926 16:42:36.131554 (+ 177us) client_negotiation.cc:240] Negotiation successful
0926 16:42:36.131851 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":113,"threads_started":1}
I20260926 16:42:36.134111 27457 heartbeater.cc:347] Connected to a master server at 127.21.12.126:45787
I20260926 16:42:36.134398 27457 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:36.134891 27457 heartbeater.cc:511] Master 127.21.12.126:45787 requested a full tablet report, sending...
I20260926 16:42:36.135982 27132 ts_manager.cc:194] Registered new tserver with Master: ecdc74d641cd47cc8056f74101335359 (127.21.12.66:42477)
I20260926 16:42:36.136780 27132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.66@KRBTEST.COM'} at 127.21.12.66:35465
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab.
Entry for principal kudu/127.21.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab.
I20260926 16:42:36.145401 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.67:0
--local_ip_for_outbound_sockets=127.21.12.67
--webserver_interface=127.21.12.67
--webserver_port=0
--tserver_master_addrs=127.21.12.126:45787
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab
--principal=kudu/127.21.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:44671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:36.252558 27462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:36.252791 27462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:36.252842 27462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:36.256806 27462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:36.256858 27462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:36.256953 27462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.67
I20260926 16:42:36.262383 27462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:44671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab
--principal=kudu/127.21.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.67
--webserver_port=0
--tserver_master_addrs=127.21.12.126:45787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.67
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:36.263317 27462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:36.264268 27462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:36.273554 27470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:36.273562 27468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:36.273562 27467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:36.273960 27462 server_base.cc:1034] running on GCE node
I20260926 16:42:36.274345 27462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:36.274994 27462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:36.276188 27462 hybrid_clock.cc:648] HybridClock initialized: now 1790440956276167 us; error 35 us; skew 500 ppm
Sep 26 16:42:36 dist-test-slave-n1hw krb5kdc[27087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440956, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:36.279117 27462 init.cc:377] Logged in from keytab as kudu/127.21.12.67@KRBTEST.COM (short username kudu)
I20260926 16:42:36.279898 27462 webserver.cc:460] Webserver started at http://127.21.12.67:43653/ using document root <none> and password file <none>
I20260926 16:42:36.280470 27462 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:36.280529 27462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:36.280784 27462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:36.282760 27462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-2/data/instance:
uuid: "0918cb29a22943988d7209f04f580f40"
format_stamp: "Formatted at 2026-09-26 16:42:36 on dist-test-slave-n1hw"
server_key: "8fedb7f841d2499d2e787708d94273c2"
server_key_iv: "35d646c6e2156bab0fcf925346a7d0dd"
server_key_version: "encryptionkey@0"
I20260926 16:42:36.283300 27462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance:
uuid: "0918cb29a22943988d7209f04f580f40"
format_stamp: "Formatted at 2026-09-26 16:42:36 on dist-test-slave-n1hw"
server_key: "8fedb7f841d2499d2e787708d94273c2"
server_key_iv: "35d646c6e2156bab0fcf925346a7d0dd"
server_key_version: "encryptionkey@0"
I20260926 16:42:36.286752 27462 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:42:36.288998 27477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:36.290271 27462 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:42:36.290385 27462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-2/wal
uuid: "0918cb29a22943988d7209f04f580f40"
format_stamp: "Formatted at 2026-09-26 16:42:36 on dist-test-slave-n1hw"
server_key: "8fedb7f841d2499d2e787708d94273c2"
server_key_iv: "35d646c6e2156bab0fcf925346a7d0dd"
server_key_version: "encryptionkey@0"
I20260926 16:42:36.290531 27462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:36.314507 27462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:36.317855 27462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:36.318058 27462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:36.318643 27462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:42:36.319450 27462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:36.319500 27462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:36.319567 27462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:36.319610 27462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:36.328882 27462 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.67:34123
I20260926 16:42:36.328914 27590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.67:34123 every 8 connection(s)
I20260926 16:42:36.329871 27462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
I20260926 16:42:36.331295 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 27462
I20260926 16:42:36.331460 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance
I20260926 16:42:36.331790 21553 external_mini_cluster.cc:1428] Setting key a5c79dd26bf863b704525d22f36859e8
Sep 26 16:42:36 dist-test-slave-n1hw krb5kdc[27087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440956, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.67@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:42:36.342139 27171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:36.331604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:37957 (local address 127.21.12.126:45787)
0926 16:42:36.331759 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:36.331764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:36.332699 (+ 935us) server_negotiation.cc:407] Connection header received
0926 16:42:36.334069 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:36.334071 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:36.334114 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:36.334180 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:36.335208 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:36.335716 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:36.336758 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:36.336969 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:36.337047 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:36.339210 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:36.339213 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:36.339215 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:36.339236 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:36.340443 (+ 1207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:36.340872 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:36.340874 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:36.340875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:36.340945 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:36.341217 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:36.341219 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:36.341221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:36.341375 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:36.341447 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:36.341924 (+ 477us) server_negotiation.cc:299] Negotiation successful
0926 16:42:36.342024 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 16:42:36.342890 27593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:36.332082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:45787 (local address 127.21.12.67:37957)
0926 16:42:36.332558 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:36.332612 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:36.333890 (+ 1278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:36.334310 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:36.334316 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:36.334646 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:36.335091 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:36.335104 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:36.335818 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:36.335821 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:36.336658 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:36.336666 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:36.337089 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:36.337091 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:36.337222 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:36.337694 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:36.337708 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:36.339067 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:36.340540 (+ 1473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:36.340544 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:36.340554 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:36.340790 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:36.341020 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:36.341022 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:36.341024 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:36.341143 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:36.341463 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:36.341467 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:36.341668 (+ 201us) client_negotiation.cc:769] Sending connection context
0926 16:42:36.341880 (+ 212us) client_negotiation.cc:240] Negotiation successful
0926 16:42:36.342110 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":136,"threads_started":1}
I20260926 16:42:36.343904 27591 heartbeater.cc:347] Connected to a master server at 127.21.12.126:45787
I20260926 16:42:36.344148 27591 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:36.344635 27591 heartbeater.cc:511] Master 127.21.12.126:45787 requested a full tablet report, sending...
I20260926 16:42:36.345583 27132 ts_manager.cc:194] Registered new tserver with Master: 0918cb29a22943988d7209f04f580f40 (127.21.12.67:34123)
I20260926 16:42:36.346174 27132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.67@KRBTEST.COM'} at 127.21.12.67:37957
I20260926 16:42:36.357460 21553 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/krb5kdc/test-admin.keytab.
Sep 26 16:42:36 dist-test-slave-n1hw krb5kdc[27087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:36.374416 21553 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 16:42:36.377140 27598 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:36.375633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34754 (local address 127.21.12.65:34555)
0926 16:42:36.375952 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:36.375971 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:36.375999 (+ 28us) server_negotiation.cc:407] Connection header received
0926 16:42:36.376058 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:36.376063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:36.376242 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:36.376393 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:36.376953 (+ 560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34754: BlockingRecv error: recv got EOF from 127.0.0.1:34754 (error 108)
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":112,"threads_started":1}
W20260926 16:42:36.377250 21553 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/krb5kdc/krb5cc' of type FILE
Sep 26 16:42:36 dist-test-slave-n1hw krb5kdc[27087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:36.378309 21553 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 16:42:36 dist-test-slave-n1hw krb5kdc[27087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:36.379295 21553 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 16:42:36 dist-test-slave-n1hw krb5kdc[27087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.65@KRBTEST.COM
I20260926 16:42:36.388433 27598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:36.380118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34758 (local address 127.21.12.65:34555)
0926 16:42:36.380215 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:36.380219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:36.380281 (+ 62us) server_negotiation.cc:407] Connection header received
0926 16:42:36.380370 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:36.380372 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:36.380409 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:36.380474 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:36.380937 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:36.381696 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:36.382560 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:36.382912 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:36.383027 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:36.385165 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:36.385173 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:36.385185 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:36.385206 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:36.386792 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:36.387222 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:36.387227 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:36.387228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:36.387301 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:36.387552 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:36.387554 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:36.387555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:36.387853 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:36.388003 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:36.388205 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 16:42:36.388318 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260926 16:42:36.392145 27598 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:36.391221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34762 (local address 127.21.12.65:34555)
0926 16:42:36.391323 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:36.391326 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:36.391508 (+ 182us) server_negotiation.cc:407] Connection header received
0926 16:42:36.391596 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:36.391598 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:36.391630 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:36.391694 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:36.392035 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34762: BlockingRecv error: recv got EOF from 127.0.0.1:34762 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260926 16:42:36.392418 21553 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/krb5kdc/krb5cc' of type FILE
Sep 26 16:42:36 dist-test-slave-n1hw krb5kdc[27087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:36.393348 21553 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 16:42:36 dist-test-slave-n1hw krb5kdc[27087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:36.394322 21553 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 16:42:36 dist-test-slave-n1hw krb5kdc[27087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.65@KRBTEST.COM
I20260926 16:42:36.402798 27598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:36.395107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34770 (local address 127.21.12.65:34555)
0926 16:42:36.395243 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:36.395246 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:36.395277 (+ 31us) server_negotiation.cc:407] Connection header received
0926 16:42:36.395384 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:36.395387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:36.395421 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:36.395492 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:36.395915 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:36.396430 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:36.397151 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:36.397384 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:36.397470 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:36.399542 (+ 2072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:36.399545 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:36.399547 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:36.399567 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:36.400968 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:36.401492 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:36.401496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:36.401497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:36.401592 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:36.401918 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:36.401924 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:36.401926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:36.402137 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:36.402264 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:36.402549 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 16:42:36.402655 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 16:42:36.408890 27598 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:36.406805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34776 (local address 127.21.12.65:34555)
0926 16:42:36.406905 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:36.406908 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:36.408140 (+ 1232us) server_negotiation.cc:407] Connection header received
0926 16:42:36.408213 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:36.408215 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:36.408279 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:36.408436 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:36.408821 (+ 385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34776: BlockingRecv error: recv got EOF from 127.0.0.1:34776 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260926 16:42:36.409369 21553 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440924937351-21553-0/krb5kdc/krb5cc' of type FILE
Sep 26 16:42:36 dist-test-slave-n1hw krb5kdc[27087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:36.410456 21553 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 16:42:36 dist-test-slave-n1hw krb5kdc[27087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:36.411363 21553 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 16:42:36 dist-test-slave-n1hw krb5kdc[27087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.65@KRBTEST.COM
I20260926 16:42:36.419878 27598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:36.412085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34784 (local address 127.21.12.65:34555)
0926 16:42:36.412227 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:36.412232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:36.412289 (+ 57us) server_negotiation.cc:407] Connection header received
0926 16:42:36.412401 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:36.412405 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:36.412448 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:36.412527 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:36.413086 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:36.413825 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:36.414561 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:36.414759 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:36.414836 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:36.416912 (+ 2076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:36.416916 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:36.416919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:36.416939 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:36.418407 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:36.418764 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:36.418766 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:36.418768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:36.418833 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:36.419087 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:36.419090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:36.419091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:36.419253 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:36.419410 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:36.419575 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 16:42:36.419697 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 16:42:36.420948 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 27194
I20260926 16:42:36.426667 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 27328
I20260926 16:42:36.431902 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 27462
I20260926 16:42:36.436896 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 27100
2026-09-26T16:42:36Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2147 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:42:36 dist-test-slave-n1hw krb5kdc[27611](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:42:36 dist-test-slave-n1hw krb5kdc[27611](info): set up 2 sockets
Sep 26 16:42:36 dist-test-slave-n1hw krb5kdc[27611](info): commencing operation
krb5kdc: starting...
W20260926 16:42:37.484179 21553 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Sep 26 16:42:37 dist-test-slave-n1hw krb5kdc[27611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:42:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:42:37Z Disabled control of system clock
WARNING: no policy specified for oryx/127.21.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.126.keytab.
Entry for principal oryx/127.21.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.126.keytab.
I20260926 16:42:37.604496 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:34799
--webserver_interface=127.21.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.126.keytab
--principal=oryx/127.21.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:38299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:34799
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:37.710528 27624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:37.710774 27624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:37.710814 27624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:37.714766 27624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:42:37.714816 27624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:37.714835 27624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:37.714856 27624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:42:37.714913 27624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:42:37.720108 27624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:38299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:34799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.126.keytab
--principal=oryx/127.21.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:34799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:37.721077 27624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:37.722074 27624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:37.730235 27630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:37.730293 27632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:37.730242 27629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:37.730355 27624 server_base.cc:1034] running on GCE node
I20260926 16:42:37.731114 27624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:37.732149 27624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:37.733340 27624 hybrid_clock.cc:648] HybridClock initialized: now 1790440957733314 us; error 35 us; skew 500 ppm
Sep 26 16:42:37 dist-test-slave-n1hw krb5kdc[27611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440957, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:37.736044 27624 init.cc:377] Logged in from keytab as oryx/127.21.12.126@KRBTEST.COM (short username oryx)
I20260926 16:42:37.736829 27624 webserver.cc:460] Webserver started at http://127.21.12.126:39389/ using document root <none> and password file <none>
I20260926 16:42:37.737354 27624 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:37.737442 27624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:37.737676 27624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:37.739624 27624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "39e91c6cfaa7419891443717312b224d"
format_stamp: "Formatted at 2026-09-26 16:42:37 on dist-test-slave-n1hw"
server_key: "daa4bd3fbb762f95731b158bae14d07b"
server_key_iv: "08a3c5dacbb9e7311247433a20fb33de"
server_key_version: "encryptionkey@0"
I20260926 16:42:37.740161 27624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "39e91c6cfaa7419891443717312b224d"
format_stamp: "Formatted at 2026-09-26 16:42:37 on dist-test-slave-n1hw"
server_key: "daa4bd3fbb762f95731b158bae14d07b"
server_key_iv: "08a3c5dacbb9e7311247433a20fb33de"
server_key_version: "encryptionkey@0"
I20260926 16:42:37.743651 27624 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 16:42:37.745946 27639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:37.747081 27624 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:42:37.747223 27624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "39e91c6cfaa7419891443717312b224d"
format_stamp: "Formatted at 2026-09-26 16:42:37 on dist-test-slave-n1hw"
server_key: "daa4bd3fbb762f95731b158bae14d07b"
server_key_iv: "08a3c5dacbb9e7311247433a20fb33de"
server_key_version: "encryptionkey@0"
I20260926 16:42:37.747329 27624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:37.761685 27624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:37.764366 27624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:37.764564 27624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:37.771365 27624 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.126:34799
I20260926 16:42:37.771638 27691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.126:34799 every 8 connection(s)
I20260926 16:42:37.772308 27624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:42:37.774987 27692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:37.780020 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 27624
I20260926 16:42:37.780189 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:42:37.780388 27692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d: Bootstrap starting.
I20260926 16:42:37.780495 21553 external_mini_cluster.cc:1428] Setting key f08e9715915c05bf59313fa1843efa51
I20260926 16:42:37.782867 27692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:37.783576 27692 log.cc:824] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:37.785322 27692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d: No bootstrap required, opened a new log
I20260926 16:42:37.788980 27692 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } }
Sep 26 16:42:37 dist-test-slave-n1hw krb5kdc[27611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM
I20260926 16:42:37.789263 27692 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:37.789314 27692 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39e91c6cfaa7419891443717312b224d, State: Initialized, Role: FOLLOWER
I20260926 16:42:37.789814 27692 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } }
I20260926 16:42:37.789974 27692 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:42:37.790040 27692 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:42:37.790130 27692 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:37.791195 27692 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } }
I20260926 16:42:37.791522 27692 leader_election.cc:302] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39e91c6cfaa7419891443717312b224d; no voters:
I20260926 16:42:37.791823 27692 leader_election.cc:288] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:42:37.791903 27697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:37.792141 27697 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d [term 1 LEADER]: Becoming Leader. State: Replica: 39e91c6cfaa7419891443717312b224d, State: Running, Role: LEADER
I20260926 16:42:37.792508 27697 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } }
I20260926 16:42:37.793172 27692 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:42:37.794178 27698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39e91c6cfaa7419891443717312b224d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } }
I20260926 16:42:37.794425 27698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:37.794802 27706 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:42:37.795219 27695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:37.781826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56960 (local address 127.21.12.126:34799)
0926 16:42:37.782314 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:37.782330 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:37.782365 (+ 35us) server_negotiation.cc:407] Connection header received
0926 16:42:37.783069 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:37.783086 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:37.783417 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:37.783757 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:37.784586 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:37.785615 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:37.786476 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:37.786932 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:37.787099 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:37.789913 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:37.789918 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:37.789929 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:37.789974 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:37.791832 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:37.792371 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:37.792377 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:37.792382 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:37.792528 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:37.792869 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:37.792872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:37.792874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:37.793354 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:37.793573 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:37.793961 (+ 388us) server_negotiation.cc:299] Negotiation successful
0926 16:42:37.794274 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":131,"threads_started":1}
I20260926 16:42:37.795424 27699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39e91c6cfaa7419891443717312b224d. Latest consensus state: current_term: 1 leader_uuid: "39e91c6cfaa7419891443717312b224d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } }
I20260926 16:42:37.795516 27699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:37.797683 27706 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:42:37.803335 27706 catalog_manager.cc:1347] Generated new cluster ID: c302d5e480d84aabadda95d4bba72d12
I20260926 16:42:37.803404 27706 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:42:37.814092 27706 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:42:37.815922 27706 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:42:37.821977 27706 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d: Generated new TSK 0
I20260926 16:42:37.822714 27706 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.21.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.65.keytab.
Entry for principal oryx/127.21.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.65.keytab.
I20260926 16:42:37.854804 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.65:0
--local_ip_for_outbound_sockets=127.21.12.65
--webserver_interface=127.21.12.65
--webserver_port=0
--tserver_master_addrs=127.21.12.126:34799
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.65.keytab
--principal=oryx/127.21.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:38299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:37.963766 27718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:37.964036 27718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:37.964079 27718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:37.968094 27718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:37.968158 27718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:37.968247 27718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.65
I20260926 16:42:37.973574 27718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:38299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.65.keytab
--principal=oryx/127.21.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.65
--webserver_port=0
--tserver_master_addrs=127.21.12.126:34799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.65
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:37.974689 27718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:37.975741 27718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:37.985628 27723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:37.985636 27726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:37.985703 27718 server_base.cc:1034] running on GCE node
W20260926 16:42:37.985644 27724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:37.986207 27718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:37.986831 27718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:37.987989 27718 hybrid_clock.cc:648] HybridClock initialized: now 1790440957987971 us; error 29 us; skew 500 ppm
Sep 26 16:42:37 dist-test-slave-n1hw krb5kdc[27611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440957, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:37.990725 27718 init.cc:377] Logged in from keytab as oryx/127.21.12.65@KRBTEST.COM (short username oryx)
I20260926 16:42:37.991504 27718 webserver.cc:460] Webserver started at http://127.21.12.65:39041/ using document root <none> and password file <none>
I20260926 16:42:37.992004 27718 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:37.992062 27718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:37.992216 27718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:37.994437 27718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-0/data/instance:
uuid: "2d57671a949840679f2e9b9ec4ddb8de"
format_stamp: "Formatted at 2026-09-26 16:42:37 on dist-test-slave-n1hw"
server_key: "24add1f576cd6a08a45428d5a0557c98"
server_key_iv: "0b041479eeeb36a4d0f29a7dda71dd3f"
server_key_version: "encryptionkey@0"
I20260926 16:42:37.995007 27718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance:
uuid: "2d57671a949840679f2e9b9ec4ddb8de"
format_stamp: "Formatted at 2026-09-26 16:42:37 on dist-test-slave-n1hw"
server_key: "24add1f576cd6a08a45428d5a0557c98"
server_key_iv: "0b041479eeeb36a4d0f29a7dda71dd3f"
server_key_version: "encryptionkey@0"
I20260926 16:42:37.998750 27718 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 16:42:38.001118 27733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:38.002475 27718 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:42:38.002614 27718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-0/wal
uuid: "2d57671a949840679f2e9b9ec4ddb8de"
format_stamp: "Formatted at 2026-09-26 16:42:37 on dist-test-slave-n1hw"
server_key: "24add1f576cd6a08a45428d5a0557c98"
server_key_iv: "0b041479eeeb36a4d0f29a7dda71dd3f"
server_key_version: "encryptionkey@0"
I20260926 16:42:38.002731 27718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:38.022251 27718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:38.025316 27718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:38.025537 27718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:38.026193 27718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:42:38.027048 27718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:38.027125 27718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:38.027197 27718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:38.027240 27718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:38.035785 27718 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.65:37069
I20260926 16:42:38.035815 27846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.65:37069 every 8 connection(s)
I20260926 16:42:38.036723 27718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
I20260926 16:42:38.040894 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 27718
I20260926 16:42:38.041064 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance
I20260926 16:42:38.041390 21553 external_mini_cluster.cc:1428] Setting key 0e87fbdf5ce740228e7e02ff8a7f56b2
Sep 26 16:42:38 dist-test-slave-n1hw krb5kdc[27611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440957, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.65@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.21.12.66@KRBTEST.COM; defaulting to no policy
I20260926 16:42:38.052075 27695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:38.038598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:60245 (local address 127.21.12.126:34799)
0926 16:42:38.038732 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:38.038736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:38.039613 (+ 877us) server_negotiation.cc:407] Connection header received
0926 16:42:38.041121 (+ 1508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:38.041124 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:38.041185 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:38.041288 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:38.042935 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:38.043723 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:38.044909 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:38.045149 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:38.045264 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:38.048096 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:38.048099 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:38.048101 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:38.048124 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:38.049685 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:38.050313 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:38.050318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:38.050323 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:38.050407 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:38.050845 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:38.050848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:38.050849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:38.051035 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:38.051112 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:38.051766 (+ 654us) server_negotiation.cc:299] Negotiation successful
0926 16:42:38.051873 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 16:42:38.052834 27849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:38.038967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34799 (local address 127.21.12.65:60245)
0926 16:42:38.039448 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:38.039501 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:38.040807 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:38.041520 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:38.041530 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:38.042059 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:38.042762 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:38.042782 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:38.043863 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:38.043866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:38.044783 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:38.044793 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:38.045299 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:38.045303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:38.045472 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:38.046107 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:38.046123 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:38.047913 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:38.049821 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:38.049829 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:38.049845 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:38.050197 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:38.050513 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:38.050516 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:38.050519 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:38.050711 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:38.051144 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:38.051150 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:38.051493 (+ 343us) client_negotiation.cc:769] Sending connection context
0926 16:42:38.051698 (+ 205us) client_negotiation.cc:240] Negotiation successful
0926 16:42:38.051958 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":138,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.12.66@KRBTEST.COM" created.
I20260926 16:42:38.054303 27847 heartbeater.cc:347] Connected to a master server at 127.21.12.126:34799
I20260926 16:42:38.054554 27847 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:38.055056 27847 heartbeater.cc:511] Master 127.21.12.126:34799 requested a full tablet report, sending...
I20260926 16:42:38.056540 27656 ts_manager.cc:194] Registered new tserver with Master: 2d57671a949840679f2e9b9ec4ddb8de (127.21.12.65:37069)
I20260926 16:42:38.057745 27656 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.12.65@KRBTEST.COM'} at 127.21.12.65:60245
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.66.keytab.
Entry for principal oryx/127.21.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.66.keytab.
I20260926 16:42:38.069332 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.66:0
--local_ip_for_outbound_sockets=127.21.12.66
--webserver_interface=127.21.12.66
--webserver_port=0
--tserver_master_addrs=127.21.12.126:34799
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.66.keytab
--principal=oryx/127.21.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:38299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:38.177726 27852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:38.178046 27852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:38.178143 27852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:38.182345 27852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:38.182423 27852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:38.182505 27852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.66
I20260926 16:42:38.188232 27852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:38299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.66.keytab
--principal=oryx/127.21.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.66
--webserver_port=0
--tserver_master_addrs=127.21.12.126:34799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.66
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:38.189263 27852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:38.190308 27852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:38.200145 27860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:38.200174 27858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:38.200145 27857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:38.200325 27852 server_base.cc:1034] running on GCE node
I20260926 16:42:38.200771 27852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:38.201362 27852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:38.202525 27852 hybrid_clock.cc:648] HybridClock initialized: now 1790440958202508 us; error 28 us; skew 500 ppm
Sep 26 16:42:38 dist-test-slave-n1hw krb5kdc[27611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440958, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:38.205224 27852 init.cc:377] Logged in from keytab as oryx/127.21.12.66@KRBTEST.COM (short username oryx)
I20260926 16:42:38.206202 27852 webserver.cc:460] Webserver started at http://127.21.12.66:36899/ using document root <none> and password file <none>
I20260926 16:42:38.206902 27852 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:38.206992 27852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:38.207232 27852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:38.209230 27852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-1/data/instance:
uuid: "0df678e3047b4d9ca7e774b8040f86b4"
format_stamp: "Formatted at 2026-09-26 16:42:38 on dist-test-slave-n1hw"
server_key: "024f1adfbc663ec248a4a15ecce6cf22"
server_key_iv: "c985e37548341c7d524a994afc827bef"
server_key_version: "encryptionkey@0"
I20260926 16:42:38.209849 27852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance:
uuid: "0df678e3047b4d9ca7e774b8040f86b4"
format_stamp: "Formatted at 2026-09-26 16:42:38 on dist-test-slave-n1hw"
server_key: "024f1adfbc663ec248a4a15ecce6cf22"
server_key_iv: "c985e37548341c7d524a994afc827bef"
server_key_version: "encryptionkey@0"
I20260926 16:42:38.213407 27852 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:42:38.215833 27867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:38.217023 27852 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:42:38.217170 27852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-1/wal
uuid: "0df678e3047b4d9ca7e774b8040f86b4"
format_stamp: "Formatted at 2026-09-26 16:42:38 on dist-test-slave-n1hw"
server_key: "024f1adfbc663ec248a4a15ecce6cf22"
server_key_iv: "c985e37548341c7d524a994afc827bef"
server_key_version: "encryptionkey@0"
I20260926 16:42:38.217278 27852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:38.236948 27852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:38.240231 27852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:38.240439 27852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:38.241027 27852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:42:38.241916 27852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:38.241986 27852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:38.242067 27852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:38.242110 27852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:38.250928 27852 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.66:41203
I20260926 16:42:38.250979 27980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.66:41203 every 8 connection(s)
I20260926 16:42:38.251941 27852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
I20260926 16:42:38.255600 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 27852
I20260926 16:42:38.255697 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance
I20260926 16:42:38.256031 21553 external_mini_cluster.cc:1428] Setting key 286530f5964c14e8628e8b74e6cce508
Sep 26 16:42:38 dist-test-slave-n1hw krb5kdc[27611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440958, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.66@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.21.12.67@KRBTEST.COM; defaulting to no policy
I20260926 16:42:38.267391 27695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:38.253712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:50563 (local address 127.21.12.126:34799)
0926 16:42:38.253870 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:38.253875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:38.254784 (+ 909us) server_negotiation.cc:407] Connection header received
0926 16:42:38.256247 (+ 1463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:38.256250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:38.256308 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:38.256403 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:38.257892 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:38.258439 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:38.259929 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:38.260162 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:38.260277 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:38.263304 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:38.263308 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:38.263310 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:38.263334 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:38.264907 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:38.265552 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:38.265555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:38.265556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:38.265635 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:38.266123 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:38.266125 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:38.266126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:38.266309 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:38.266387 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:38.267087 (+ 700us) server_negotiation.cc:299] Negotiation successful
0926 16:42:38.267216 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 16:42:38.268144 27983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:38.254185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34799 (local address 127.21.12.66:50563)
0926 16:42:38.254629 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:38.254674 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:38.255990 (+ 1316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:38.256609 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:38.256618 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:38.257018 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:38.257702 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:38.257723 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:38.258580 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:38.258587 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:38.259791 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:38.259801 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:38.260312 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:38.260316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:38.260488 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:38.261103 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:38.261119 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:38.263104 (+ 1985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:38.265058 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:38.265065 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:38.265084 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:38.265444 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:38.265800 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:38.265805 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:38.265808 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:38.266011 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:38.266403 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:38.266408 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:38.266789 (+ 381us) client_negotiation.cc:769] Sending connection context
0926 16:42:38.266954 (+ 165us) client_negotiation.cc:240] Negotiation successful
0926 16:42:38.267241 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":123,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.12.67@KRBTEST.COM" created.
I20260926 16:42:38.269414 27981 heartbeater.cc:347] Connected to a master server at 127.21.12.126:34799
I20260926 16:42:38.269677 27981 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:38.270320 27981 heartbeater.cc:511] Master 127.21.12.126:34799 requested a full tablet report, sending...
I20260926 16:42:38.271416 27656 ts_manager.cc:194] Registered new tserver with Master: 0df678e3047b4d9ca7e774b8040f86b4 (127.21.12.66:41203)
I20260926 16:42:38.271998 27656 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.12.66@KRBTEST.COM'} at 127.21.12.66:50563
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.67.keytab.
Entry for principal oryx/127.21.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.67.keytab.
I20260926 16:42:38.285498 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.67:0
--local_ip_for_outbound_sockets=127.21.12.67
--webserver_interface=127.21.12.67
--webserver_port=0
--tserver_master_addrs=127.21.12.126:34799
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.67.keytab
--principal=oryx/127.21.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:38299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:38.402070 27986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:38.402297 27986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:38.402338 27986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:38.406361 27986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:38.406445 27986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:38.406503 27986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.67
I20260926 16:42:38.411612 27986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:38299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.67.keytab
--principal=oryx/127.21.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.67
--webserver_port=0
--tserver_master_addrs=127.21.12.126:34799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.67
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:38.412597 27986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:38.413578 27986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:38.423122 27992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:38.423125 27991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:38.423313 27986 server_base.cc:1034] running on GCE node
W20260926 16:42:38.423125 27994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:38.423789 27986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:38.424463 27986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:38.425647 27986 hybrid_clock.cc:648] HybridClock initialized: now 1790440958425625 us; error 29 us; skew 500 ppm
Sep 26 16:42:38 dist-test-slave-n1hw krb5kdc[27611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440958, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:38.428402 27986 init.cc:377] Logged in from keytab as oryx/127.21.12.67@KRBTEST.COM (short username oryx)
I20260926 16:42:38.429337 27986 webserver.cc:460] Webserver started at http://127.21.12.67:45099/ using document root <none> and password file <none>
I20260926 16:42:38.429957 27986 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:38.430048 27986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:38.430274 27986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:38.432261 27986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-2/data/instance:
uuid: "b27e9cb575dc439eae6edd24bc55de6c"
format_stamp: "Formatted at 2026-09-26 16:42:38 on dist-test-slave-n1hw"
server_key: "88226f8de3bc232d4007ae8c93bd73fa"
server_key_iv: "87d55b0ccf41b0754991b224167cd2e9"
server_key_version: "encryptionkey@0"
I20260926 16:42:38.432822 27986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance:
uuid: "b27e9cb575dc439eae6edd24bc55de6c"
format_stamp: "Formatted at 2026-09-26 16:42:38 on dist-test-slave-n1hw"
server_key: "88226f8de3bc232d4007ae8c93bd73fa"
server_key_iv: "87d55b0ccf41b0754991b224167cd2e9"
server_key_version: "encryptionkey@0"
I20260926 16:42:38.436372 27986 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260926 16:42:38.438812 28001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:38.440006 27986 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:42:38.440142 27986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-2/wal
uuid: "b27e9cb575dc439eae6edd24bc55de6c"
format_stamp: "Formatted at 2026-09-26 16:42:38 on dist-test-slave-n1hw"
server_key: "88226f8de3bc232d4007ae8c93bd73fa"
server_key_iv: "87d55b0ccf41b0754991b224167cd2e9"
server_key_version: "encryptionkey@0"
I20260926 16:42:38.440250 27986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:38.459936 27986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:38.463268 27986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:38.463475 27986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:38.464064 27986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:42:38.464866 27986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:38.464933 27986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:38.465006 27986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:38.465050 27986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:38.473839 27986 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.67:42229
I20260926 16:42:38.473857 28114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.67:42229 every 8 connection(s)
I20260926 16:42:38.474680 27986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
I20260926 16:42:38.481745 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 27986
I20260926 16:42:38.481881 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance
I20260926 16:42:38.482197 21553 external_mini_cluster.cc:1428] Setting key a20845a7c99609076a2d84a6b99759d0
Sep 26 16:42:38 dist-test-slave-n1hw krb5kdc[27611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440958, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.67@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM
I20260926 16:42:38.487555 27695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:38.476297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:45497 (local address 127.21.12.126:34799)
0926 16:42:38.476424 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:38.476428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:38.477276 (+ 848us) server_negotiation.cc:407] Connection header received
0926 16:42:38.478564 (+ 1288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:38.478567 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:38.478613 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:38.478688 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:38.479774 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:38.480309 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:38.481549 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:38.481922 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:38.482064 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:38.484464 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:38.484467 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:38.484469 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:38.484491 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:38.485805 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:38.486256 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:38.486259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:38.486260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:38.486333 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:38.486652 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:38.486654 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:38.486655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:38.486818 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:38.486894 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:38.487315 (+ 421us) server_negotiation.cc:299] Negotiation successful
0926 16:42:38.487416 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 16:42:38.488269 28117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:38.476701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34799 (local address 127.21.12.67:45497)
0926 16:42:38.477125 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:38.477165 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:38.478372 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:38.478839 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:38.478846 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:38.479200 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:38.479635 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:38.479648 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:38.480451 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:38.480454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:38.481351 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:38.481361 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:38.482083 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:38.482086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:38.482224 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:38.482887 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:38.482902 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:38.484314 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:38.485916 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:38.485920 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:38.485931 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:38.486174 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:38.486445 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:38.486447 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:38.486449 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:38.486572 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:38.486906 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:38.486911 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:38.487114 (+ 203us) client_negotiation.cc:769] Sending connection context
0926 16:42:38.487259 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 16:42:38.487494 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":118,"threads_started":1}
I20260926 16:42:38.489531 28115 heartbeater.cc:347] Connected to a master server at 127.21.12.126:34799
I20260926 16:42:38.489718 28115 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:38.490180 28115 heartbeater.cc:511] Master 127.21.12.126:34799 requested a full tablet report, sending...
I20260926 16:42:38.491180 27656 ts_manager.cc:194] Registered new tserver with Master: b27e9cb575dc439eae6edd24bc55de6c (127.21.12.67:42229)
I20260926 16:42:38.491688 27656 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.12.67@KRBTEST.COM'} at 127.21.12.67:45497
I20260926 16:42:38.495819 21553 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.21.12.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.12.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.12.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.125.keytab.
Entry for principal oryx/127.21.12.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.125.keytab.
I20260926 16:42:38.521104 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.125:35885
--webserver_interface=127.21.12.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.125.keytab
--principal=oryx/127.21.12.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:38299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:34799,127.21.12.125:35885
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:38.628610 28120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:38.628829 28120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:38.628865 28120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:38.632793 28120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:42:38.632844 28120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:38.632865 28120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:38.632881 28120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:42:38.632896 28120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:42:38.637882 28120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:38299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:34799,127.21.12.125:35885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.125.keytab
--principal=oryx/127.21.12.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.125:35885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.12.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
W20260926 16:42:38.638037 28120 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.21.12.126:34799,127.21.12.125:35885'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 16:42:38.638780 28120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:38.639673 28120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:38.647620 28125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:38.647692 28126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:38.647617 28128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:38.648073 28120 server_base.cc:1034] running on GCE node
I20260926 16:42:38.648589 28120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:38.649616 28120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:38.650802 28120 hybrid_clock.cc:648] HybridClock initialized: now 1790440958650793 us; error 31 us; skew 500 ppm
Sep 26 16:42:38 dist-test-slave-n1hw krb5kdc[27611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440958, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:38.653489 28120 init.cc:377] Logged in from keytab as oryx/127.21.12.125@KRBTEST.COM (short username oryx)
I20260926 16:42:38.654306 28120 webserver.cc:460] Webserver started at http://127.21.12.125:38219/ using document root <none> and password file <none>
I20260926 16:42:38.654793 28120 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:38.654848 28120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:38.655000 28120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:38.656850 28120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-1/data/instance:
uuid: "1f70fdb997ce4656b970b08582ad2556"
format_stamp: "Formatted at 2026-09-26 16:42:38 on dist-test-slave-n1hw"
server_key: "9f6a5f1b661595c4f7b870c96a4b5ae4"
server_key_iv: "05e560341f634ce54a808cc4af103137"
server_key_version: "encryptionkey@0"
I20260926 16:42:38.657322 28120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-1/wal/instance:
uuid: "1f70fdb997ce4656b970b08582ad2556"
format_stamp: "Formatted at 2026-09-26 16:42:38 on dist-test-slave-n1hw"
server_key: "9f6a5f1b661595c4f7b870c96a4b5ae4"
server_key_iv: "05e560341f634ce54a808cc4af103137"
server_key_version: "encryptionkey@0"
I20260926 16:42:38.660894 28120 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 16:42:38.663209 28135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:38.664420 28120 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 16:42:38.664541 28120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-1/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-1/wal
uuid: "1f70fdb997ce4656b970b08582ad2556"
format_stamp: "Formatted at 2026-09-26 16:42:38 on dist-test-slave-n1hw"
server_key: "9f6a5f1b661595c4f7b870c96a4b5ae4"
server_key_iv: "05e560341f634ce54a808cc4af103137"
server_key_version: "encryptionkey@0"
I20260926 16:42:38.664623 28120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:38.696920 28120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:38.700354 28120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:38.700533 28120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:38.707748 28120 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.125:35885
I20260926 16:42:38.707787 28187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.125:35885 every 8 connection(s)
I20260926 16:42:38.708767 28120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-1/data/info.pb
I20260926 16:42:38.711611 28188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:38.715409 28188 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } has no permanent_uuid. Determining permanent_uuid...
I20260926 16:42:38.717118 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 28120
Sep 26 16:42:38 dist-test-slave-n1hw krb5kdc[27611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440958, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.125@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM
Sep 26 16:42:38 dist-test-slave-n1hw krb5kdc[27611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.12.125@KRBTEST.COM
I20260926 16:42:38.729594 27695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:38.717247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56962 (local address 127.21.12.126:34799)
0926 16:42:38.717383 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:38.717386 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:38.718487 (+ 1101us) server_negotiation.cc:407] Connection header received
0926 16:42:38.719525 (+ 1038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:38.719528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:38.719587 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:38.719688 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:38.720509 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:38.721124 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:38.722238 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:38.722456 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:38.722562 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:38.725131 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:38.725135 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:38.725141 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:38.725179 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:38.727015 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:38.727576 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:38.727582 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:38.727584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:38.727673 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:38.728509 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:38.728512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:38.728514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:38.728757 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:38.728871 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:38.729289 (+ 418us) server_negotiation.cc:299] Negotiation successful
0926 16:42:38.729423 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 16:42:38.730655 28190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:38.717723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34799 (local address 127.0.0.1:56962)
0926 16:42:38.718321 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:38.718382 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:38.719311 (+ 929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:38.719748 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:38.719754 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:38.720127 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:38.720367 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:38.720379 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:38.721261 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:38.721263 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:38.722105 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:38.722114 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:38.722603 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:38.722607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:38.722750 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:38.723324 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:38.723338 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:38.724975 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:38.727148 (+ 2173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:38.727155 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:38.727163 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:38.727417 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:38.728202 (+ 785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:38.728205 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:38.728206 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:38.728352 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:38.728878 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:38.728886 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:38.729042 (+ 156us) client_negotiation.cc:769] Sending connection context
0926 16:42:38.729233 (+ 191us) client_negotiation.cc:240] Negotiation successful
0926 16:42:38.729642 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":155,"threads_started":1}
I20260926 16:42:38.730660 28192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:38.718110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57954 (local address 127.21.12.125:35885)
0926 16:42:38.718493 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:38.718503 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:38.718574 (+ 71us) server_negotiation.cc:407] Connection header received
0926 16:42:38.719288 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:38.719304 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:38.719495 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:38.719691 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:38.720497 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:38.721433 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:38.722327 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:38.722694 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:38.722812 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:38.725717 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:38.725725 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:38.725738 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:38.725787 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:38.727674 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:38.728190 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:38.728195 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:38.728196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:38.728310 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:38.728644 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:38.728649 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:38.728651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:38.729036 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:38.729251 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:38.729459 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 16:42:38.729904 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":261,"server-negotiator.queue_time_us":304,"thread_start_us":123,"threads_started":1}
I20260926 16:42:38.732220 28188 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.125" port: 35885 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:42:38 dist-test-slave-n1hw krb5kdc[27611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440958, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.125@KRBTEST.COM for oryx/127.21.12.125@KRBTEST.COM
I20260926 16:42:38.740372 28190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:38.732770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.125:35885 (local address 127.0.0.1:57970)
0926 16:42:38.732890 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:38.732908 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:38.732980 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:38.733289 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:38.733291 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:38.733448 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:38.733548 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:38.733554 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:38.734371 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:38.734374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:38.734937 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:38.734946 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:38.735360 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:38.735363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:38.735440 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:38.735869 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:38.735880 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:38.737160 (+ 1280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:38.738833 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:38.738835 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:38.738837 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:38.739042 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:38.739384 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:38.739388 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:38.739390 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:38.739488 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:38.739919 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:38.739921 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:38.740004 (+ 83us) client_negotiation.cc:769] Sending connection context
0926 16:42:38.740090 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 16:42:38.740205 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 16:42:38.740386 28192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:38.732824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57970 (local address 127.21.12.125:35885)
0926 16:42:38.732938 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:38.732944 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:38.732958 (+ 14us) server_negotiation.cc:407] Connection header received
0926 16:42:38.733151 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:38.733153 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:38.733190 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:38.733256 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:38.733655 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:38.734250 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:38.735055 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:38.735269 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:38.735343 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:38.737284 (+ 1941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:38.737287 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:38.737289 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:38.737309 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:38.738734 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:38.739147 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:38.739150 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:38.739152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:38.739225 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:38.739600 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:38.739603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:38.739604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:38.739790 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:38.739889 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:38.740145 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 16:42:38.740265 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":43}
I20260926 16:42:38.743908 28188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556: Bootstrap starting.
I20260926 16:42:38.746271 28188 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:38.746989 28188 log.cc:824] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:38.748909 28188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556: No bootstrap required, opened a new log
I20260926 16:42:38.752142 28188 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } peers { permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 35885 } }
I20260926 16:42:38.752339 28188 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:38.752410 28188 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f70fdb997ce4656b970b08582ad2556, State: Initialized, Role: FOLLOWER
I20260926 16:42:38.752811 28188 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } peers { permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 35885 } }
I20260926 16:42:38.753283 28194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } peers { permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 35885 } } }
I20260926 16:42:38.753420 28194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:42:38.754020 28188 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 16:42:38.759102 28205 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 16:42:38.759187 28205 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 16:42:38.760864 28120 master_runner.cc:428] Detected that this master 1f70fdb997ce4656b970b08582ad2556 is joining an existing cluster
I20260926 16:42:38.760921 28120 master_runner.cc:432] Initiating AddMaster RPC to add 127.21.12.125:35885
I20260926 16:42:38.771850 28192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:38.763335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57978 (local address 127.21.12.125:35885)
0926 16:42:38.763417 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:38.763422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:38.763471 (+ 49us) server_negotiation.cc:407] Connection header received
0926 16:42:38.763600 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:38.763602 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:38.763634 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:38.763681 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:38.764236 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:38.764993 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:38.766196 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:38.766488 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:38.766590 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:38.768262 (+ 1672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:38.768266 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:38.768268 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:38.768291 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:38.770157 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:38.770705 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:38.770708 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:38.770710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:38.770786 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:38.771064 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:38.771067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:38.771069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:38.771254 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:38.771319 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:38.771605 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 16:42:38.771706 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 16:42:38.772359 28212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:38.762995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.125:35885 (local address 127.0.0.1:57978)
0926 16:42:38.763400 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:38.763413 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:38.763510 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:38.763806 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:38.763808 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:38.764013 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:38.764126 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:38.764132 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:38.765117 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:38.765121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:38.766069 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:38.766080 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:38.766613 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:38.766617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:38.766712 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:38.767467 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:38.767487 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:38.768130 (+ 643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:38.770266 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:38.770269 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:38.770271 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:38.770590 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:38.770877 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:38.770880 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:38.770882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:38.770972 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:38.771374 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:38.771376 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:38.771445 (+ 69us) client_negotiation.cc:769] Sending connection context
0926 16:42:38.772069 (+ 624us) client_negotiation.cc:240] Negotiation successful
0926 16:42:38.772191 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":1,"spinlock_wait_cycles":4352,"thread_start_us":67,"threads_started":1}
I20260926 16:42:38.778798 27695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:38.763044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56972 (local address 127.21.12.126:34799)
0926 16:42:38.763163 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:38.763167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:38.763397 (+ 230us) server_negotiation.cc:407] Connection header received
0926 16:42:38.763575 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:38.763581 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:38.763634 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:38.763707 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:38.764302 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:38.765079 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:38.765951 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:38.766228 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:38.766317 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:38.768255 (+ 1938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:38.768258 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:38.768261 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:38.768287 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:38.770155 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:38.770690 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:38.770693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:38.770695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:38.770784 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:38.771090 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:38.771094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:38.771098 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:38.771324 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:38.771415 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:38.778497 (+ 7082us) server_negotiation.cc:299] Negotiation successful
0926 16:42:38.778639 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 16:42:38.779419 28211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:38.762968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34799 (local address 127.0.0.1:56972)
0926 16:42:38.763320 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:38.763335 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:38.763476 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:38.763746 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:38.763749 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:38.764026 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:38.764158 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:38.764165 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:38.765179 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:38.765182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:38.765830 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:38.765840 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:38.766443 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:38.766446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:38.766528 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:38.767610 (+ 1082us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:38.767622 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:38.768121 (+ 499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:38.770281 (+ 2160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:38.770284 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:38.770286 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:38.770589 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:38.770881 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:38.770884 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:38.770886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:38.770995 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:38.778179 (+ 7184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:38.778182 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:38.778290 (+ 108us) client_negotiation.cc:769] Sending connection context
0926 16:42:38.779079 (+ 789us) client_negotiation.cc:240] Negotiation successful
0926 16:42:38.779218 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"spinlock_wait_cycles":4480,"thread_start_us":100,"threads_started":1}
Sep 26 16:42:38 dist-test-slave-n1hw krb5kdc[27611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440957, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.126@KRBTEST.COM for oryx/127.21.12.125@KRBTEST.COM
WARNING: no policy specified for oryx/127.21.12.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.12.124@KRBTEST.COM" created.
I20260926 16:42:38.796845 28192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:38.784434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57986 (local address 127.21.12.125:35885)
0926 16:42:38.786026 (+ 1592us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:38.786030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:38.786219 (+ 189us) server_negotiation.cc:407] Connection header received
0926 16:42:38.786434 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:38.786437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:38.786497 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:38.786574 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:38.787406 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:38.788270 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:38.789576 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:38.790016 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:38.790133 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:38.792984 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:38.792991 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:38.792994 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:38.793019 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:38.794835 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:38.795439 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:38.795443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:38.795445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:38.795537 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:38.795859 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:38.795862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:38.795864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:38.796067 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:38.796163 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:38.796576 (+ 413us) server_negotiation.cc:299] Negotiation successful
0926 16:42:38.796687 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1516}
I20260926 16:42:38.796978 28213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:38.783763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.125:35885 (local address 127.0.0.1:57986)
0926 16:42:38.786080 (+ 2317us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:38.786135 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:38.786280 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:38.786628 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:38.786635 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:38.787117 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:38.787281 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:38.787292 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:38.788401 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:38.788404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:38.789425 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:38.789436 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:38.790138 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:38.790142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:38.790252 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:38.790856 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:38.790871 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:38.792793 (+ 1922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:38.794970 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:38.794981 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:38.794989 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:38.795326 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:38.795643 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:38.795647 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:38.795648 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:38.795758 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:38.796183 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:38.796193 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:38.796346 (+ 153us) client_negotiation.cc:769] Sending connection context
0926 16:42:38.796513 (+ 167us) client_negotiation.cc:240] Negotiation successful
0926 16:42:38.796740 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2115,"thread_start_us":86,"threads_started":1}
I20260926 16:42:38.798424 27656 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.21.12.125:35885: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "1f70fdb997ce4656b970b08582ad2556"
member_type: NON_VOTER
last_known_addr {
host: "127.21.12.125"
port: 35885
}
attrs {
promote: true
}
}
dest_uuid: "39e91c6cfaa7419891443717312b224d"
cas_config_opid_index: -1
I20260926 16:42:38.799155 27656 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } peers { permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: NON_VOTER last_known_addr { host: "127.21.12.125" port: 35885 } attrs { promote: true } }
I20260926 16:42:38.801097 28216 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 1f70fdb997ce4656b970b08582ad2556 (127.21.12.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } peers { permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: NON_VOTER last_known_addr { host: "127.21.12.125" port: 35885 } attrs { promote: true } } }
I20260926 16:42:38.802448 28162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:38.803047 28216 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.21.12.125:35885
I20260926 16:42:38.803285 28216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "39e91c6cfaa7419891443717312b224d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } peers { permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: NON_VOTER last_known_addr { host: "127.21.12.125" port: 35885 } attrs { promote: true } } }
I20260926 16:42:38.803440 28216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:38.804036 28162 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [term 1 FOLLOWER]: Refusing update from remote peer 39e91c6cfaa7419891443717312b224d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 16:42:38.804697 28120 master.cc:537] Master@127.21.12.125:35885 shutting down...
I20260926 16:42:38.805178 28217 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: NON_VOTER last_known_addr { host: "127.21.12.125" port: 35885 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 16:42:38.808270 27641 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260926 16:42:38.809324 27641 consensus_peers.cc:487] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d -> Peer 1f70fdb997ce4656b970b08582ad2556 (127.21.12.125:35885): Couldn't send request to peer 1f70fdb997ce4656b970b08582ad2556. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 16:42:38.809386 28120 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:42:38.809700 28120 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:42:38.809854 28120 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.12.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.124.keytab.
Entry for principal oryx/127.21.12.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.124.keytab.
I20260926 16:42:38.814890 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.124:42743
--webserver_interface=127.21.12.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.124.keytab
--principal=oryx/127.21.12.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:38299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:34799,127.21.12.125:35885,127.21.12.124:42743
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:42:38.824949 28120 master.cc:552] Master@127.21.12.125:35885 shutdown complete.
I20260926 16:42:38.825024 28120 master_runner.cc:305] Clearing existing system tablet
I20260926 16:42:38.825253 28120 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:38.825321 28120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:38.826637 28120 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 16:42:38.827446 28223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:38.827687 28120 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:38.827836 28120 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 16:42:38.827872 28120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-1/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-1/wal
uuid: "1f70fdb997ce4656b970b08582ad2556"
format_stamp: "Formatted at 2026-09-26 16:42:38 on dist-test-slave-n1hw"
server_key: "9f6a5f1b661595c4f7b870c96a4b5ae4"
server_key_iv: "05e560341f634ce54a808cc4af103137"
server_key_version: "encryptionkey@0"
I20260926 16:42:38.829591 28120 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:42:38.831477 28120 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 16:42:38.831578 28120 log.cc:1196] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 16:42:38.831802 28120 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556: Deleting consensus metadata
I20260926 16:42:38.831987 28120 master_runner.cc:315] Copying system tablet from 127.21.12.126:34799
I20260926 16:42:38.833120 28120 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556: tablet copy: Beginning tablet copy session from remote peer at address 127.21.12.126:34799
I20260926 16:42:38.845336 27695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:38.833832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56988 (local address 127.21.12.126:34799)
0926 16:42:38.833966 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:38.833971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:38.838027 (+ 4056us) server_negotiation.cc:407] Connection header received
0926 16:42:38.838154 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:38.838157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:38.838207 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:38.838301 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:38.838771 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:38.839399 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:38.840228 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:38.840551 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:38.840655 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:38.841910 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:38.841913 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:38.841916 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:38.841942 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:38.843565 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:38.844071 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:38.844074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:38.844076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:38.844167 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:38.844467 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:38.844471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:38.844473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:38.844651 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:38.844778 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:38.845075 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 16:42:38.845182 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 16:42:38.845347 28229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:38.833678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34799 (local address 127.0.0.1:56988)
0926 16:42:38.837916 (+ 4238us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:38.837928 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:38.838065 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:38.838317 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:38.838320 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:38.838546 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:38.838647 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:38.838652 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:38.839509 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:38.839514 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:38.840106 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:38.840114 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:38.840782 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:38.840793 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:38.840864 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:38.841408 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:38.841430 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:38.841743 (+ 313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:38.843707 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:38.843711 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:38.843716 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:38.843957 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:38.844282 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:38.844284 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:38.844286 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:38.844389 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:38.844766 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:38.844769 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:38.844871 (+ 102us) client_negotiation.cc:769] Sending connection context
0926 16:42:38.844989 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 16:42:38.845130 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4173,"thread_start_us":144,"threads_started":1}
I20260926 16:42:38.846184 27676 tablet_copy_service.cc:140] P 39e91c6cfaa7419891443717312b224d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 1f70fdb997ce4656b970b08582ad2556 ({username='oryx', principal='oryx/127.21.12.125@KRBTEST.COM'} at 127.0.0.1:56988)
I20260926 16:42:38.846355 27676 tablet_copy_service.cc:161] P 39e91c6cfaa7419891443717312b224d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 1f70fdb997ce4656b970b08582ad2556 at {username='oryx', principal='oryx/127.21.12.125@KRBTEST.COM'} at 127.0.0.1:56988: session id = 1f70fdb997ce4656b970b08582ad2556-00000000000000000000000000000000
I20260926 16:42:38.847695 27676 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 16:42:38.849144 28120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 16:42:38.851282 28120 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556: tablet copy: Starting download of 0 data blocks...
I20260926 16:42:38.851486 28120 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556: tablet copy: Starting download of 1 WAL segments...
I20260926 16:42:38.853035 28120 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 16:42:38.854598 27676 tablet_copy_service.cc:342] P 39e91c6cfaa7419891443717312b224d: Request end of tablet copy session 1f70fdb997ce4656b970b08582ad2556-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.21.12.125@KRBTEST.COM'} at 127.0.0.1:56988
I20260926 16:42:38.854722 27676 tablet_copy_service.cc:433] P 39e91c6cfaa7419891443717312b224d: ending tablet copy session 1f70fdb997ce4656b970b08582ad2556-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 1f70fdb997ce4656b970b08582ad2556
I20260926 16:42:38.861663 28120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:38.863355 28233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:38.863350 28230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:38.863638 28231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:38.865931 28120 server_base.cc:1034] running on GCE node
I20260926 16:42:38.866094 28120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:38.866420 28120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:38.867584 28120 hybrid_clock.cc:648] HybridClock initialized: now 1790440958867583 us; error 49 us; skew 500 ppm
Sep 26 16:42:38 dist-test-slave-n1hw krb5kdc[27611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440958, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:38.880549 28120 init.cc:377] Logged in from keytab as oryx/127.21.12.125@KRBTEST.COM (short username oryx)
I20260926 16:42:38.881110 28120 webserver.cc:460] Webserver started at http://127.21.12.125:43873/ using document root <none> and password file <none>
I20260926 16:42:38.881323 28120 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:38.881378 28120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:38.882620 28120 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 16:42:38.883541 28240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:38.883772 28120 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 16:42:38.883867 28120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-1/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-1/wal
uuid: "1f70fdb997ce4656b970b08582ad2556"
format_stamp: "Formatted at 2026-09-26 16:42:38 on dist-test-slave-n1hw"
server_key: "9f6a5f1b661595c4f7b870c96a4b5ae4"
server_key_iv: "05e560341f634ce54a808cc4af103137"
server_key_version: "encryptionkey@0"
I20260926 16:42:38.883968 28120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:38.889914 28120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:38.891866 28120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:38.896839 28120 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.125:35885
I20260926 16:42:38.897010 28292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.125:35885 every 8 connection(s)
I20260926 16:42:38.897432 28120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-1/data/info.pb
I20260926 16:42:38.899168 28293 sys_catalog.cc:263] Verifying existing consensus state
I20260926 16:42:38.900184 28293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556: Bootstrap starting.
I20260926 16:42:38.910506 28293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 16:42:38.910859 28293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556: Bootstrap complete.
I20260926 16:42:38.911453 28293 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } peers { permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: NON_VOTER last_known_addr { host: "127.21.12.125" port: 35885 } attrs { promote: true } }
I20260926 16:42:38.911607 28293 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1f70fdb997ce4656b970b08582ad2556, State: Initialized, Role: LEARNER
I20260926 16:42:38.911724 28293 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } peers { permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: NON_VOTER last_known_addr { host: "127.21.12.125" port: 35885 } attrs { promote: true } }
I20260926 16:42:38.912073 28296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } peers { permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: NON_VOTER last_known_addr { host: "127.21.12.125" port: 35885 } attrs { promote: true } } }
I20260926 16:42:38.912214 28293 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:42:38.912243 28296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [sys.catalog]: This master's current role is: LEARNER
I20260926 16:42:38.915150 28307 catalog_manager.cc:1259] Loaded cluster ID: c302d5e480d84aabadda95d4bba72d12
I20260926 16:42:38.915259 28307 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556: loading cluster ID for follower catalog manager: success
I20260926 16:42:38.917429 28307 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556: acquiring CA information for follower catalog manager: success
I20260926 16:42:38.918898 28307 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 16:42:38.953477 28221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:38.953712 28221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:38.953805 28221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:38.957710 28221 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:42:38.957813 28221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:38.957851 28221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:38.957883 28221 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:42:38.957913 28221 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:42:38.963161 28221 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:38299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:34799,127.21.12.125:35885,127.21.12.124:42743
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.124.keytab
--principal=oryx/127.21.12.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.124:42743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.12.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:38.964097 28221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:38.965026 28221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:38.973148 28313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:38.973140 28312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:38.973140 28315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:38.973493 28221 server_base.cc:1034] running on GCE node
I20260926 16:42:38.974074 28221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:38.975091 28221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:38.976245 28221 hybrid_clock.cc:648] HybridClock initialized: now 1790440958976223 us; error 37 us; skew 500 ppm
Sep 26 16:42:38 dist-test-slave-n1hw krb5kdc[27611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440958, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:38.979166 28221 init.cc:377] Logged in from keytab as oryx/127.21.12.124@KRBTEST.COM (short username oryx)
I20260926 16:42:38.980011 28221 webserver.cc:460] Webserver started at http://127.21.12.124:40449/ using document root <none> and password file <none>
I20260926 16:42:38.980577 28221 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:38.980639 28221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:38.980846 28221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:38.982968 28221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-2/data/instance:
uuid: "324ff229b53c4d59b8116721df0529be"
format_stamp: "Formatted at 2026-09-26 16:42:38 on dist-test-slave-n1hw"
server_key: "937bd32e3c38cfad84c86c0d0bebf3eb"
server_key_iv: "fded54cb6ebf68b121d50a99af963889"
server_key_version: "encryptionkey@0"
I20260926 16:42:38.983531 28221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-2/wal/instance:
uuid: "324ff229b53c4d59b8116721df0529be"
format_stamp: "Formatted at 2026-09-26 16:42:38 on dist-test-slave-n1hw"
server_key: "937bd32e3c38cfad84c86c0d0bebf3eb"
server_key_iv: "fded54cb6ebf68b121d50a99af963889"
server_key_version: "encryptionkey@0"
I20260926 16:42:38.987389 28221 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 16:42:38.989931 28322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:38.991153 28221 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:42:38.991288 28221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-2/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-2/wal
uuid: "324ff229b53c4d59b8116721df0529be"
format_stamp: "Formatted at 2026-09-26 16:42:38 on dist-test-slave-n1hw"
server_key: "937bd32e3c38cfad84c86c0d0bebf3eb"
server_key_iv: "fded54cb6ebf68b121d50a99af963889"
server_key_version: "encryptionkey@0"
I20260926 16:42:38.991396 28221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:39.002848 28221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:39.006058 28221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:39.006258 28221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:39.013466 28221 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.124:42743
I20260926 16:42:39.013856 28374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.124:42743 every 8 connection(s)
I20260926 16:42:39.014535 28221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-2/data/info.pb
I20260926 16:42:39.017207 28375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:39.020943 28375 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } has no permanent_uuid. Determining permanent_uuid...
I20260926 16:42:39.021988 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 28221
Sep 26 16:42:39 dist-test-slave-n1hw krb5kdc[27611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440958, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.124@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM
Sep 26 16:42:39 dist-test-slave-n1hw krb5kdc[27611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.12.124@KRBTEST.COM
I20260926 16:42:39.034407 27695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.022977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56990 (local address 127.21.12.126:34799)
0926 16:42:39.023092 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:39.023096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:39.023962 (+ 866us) server_negotiation.cc:407] Connection header received
0926 16:42:39.024750 (+ 788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:39.024753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:39.024802 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:39.024874 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:39.025939 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.026449 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.027507 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.027732 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.027809 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.030582 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:39.030586 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:39.030591 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:39.030621 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:39.032207 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:39.032783 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:39.032785 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:39.032787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:39.032856 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:39.033204 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:39.033208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:39.033210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:39.033416 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:39.033532 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:39.034115 (+ 583us) server_negotiation.cc:299] Negotiation successful
0926 16:42:39.034239 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 16:42:39.035452 28378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.023277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48058 (local address 127.21.12.124:42743)
0926 16:42:39.023760 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:39.023780 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:39.023816 (+ 36us) server_negotiation.cc:407] Connection header received
0926 16:42:39.024542 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:39.024563 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:39.024790 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:39.024968 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:39.025675 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.026713 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.027526 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.027929 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.028062 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.030970 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:39.030980 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:39.030994 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:39.031031 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:39.032886 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:39.033321 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:39.033326 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:39.033327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:39.033389 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:39.033700 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:39.033703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:39.033704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:39.033966 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:39.034156 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:39.034395 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 16:42:39.034521 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":378,"spinlock_wait_cycles":24448,"thread_start_us":161,"threads_started":1}
I20260926 16:42:39.035481 28377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.023277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34799 (local address 127.0.0.1:56990)
0926 16:42:39.023809 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:39.023856 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:39.024542 (+ 686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:39.024941 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:39.024946 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:39.025401 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:39.025785 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:39.025802 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.026567 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.026571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:39.027386 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:39.027394 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.027861 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.027865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:39.028060 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.028810 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:39.028824 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:39.030420 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:39.032317 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:39.032323 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:39.032332 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:39.032689 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:39.032945 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:39.032949 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:39.032952 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:39.033108 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:39.033558 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:39.033562 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:39.033859 (+ 297us) client_negotiation.cc:769] Sending connection context
0926 16:42:39.034056 (+ 197us) client_negotiation.cc:240] Negotiation successful
0926 16:42:39.034404 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":94,"spinlock_wait_cycles":5504,"thread_start_us":149,"threads_started":1}
I20260926 16:42:39.036861 28375 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.125" port: 35885 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:42:39 dist-test-slave-n1hw krb5kdc[27611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440958, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.124@KRBTEST.COM for oryx/127.21.12.125@KRBTEST.COM
I20260926 16:42:39.045593 28377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.037344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.125:35885 (local address 127.0.0.1:57992)
0926 16:42:39.037593 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:39.037603 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:39.037660 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:39.038392 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:39.038394 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:39.038514 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:39.038605 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:39.038609 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.039869 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.039871 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:39.040381 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:39.040388 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.040834 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.040841 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:39.040942 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.041399 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:39.041419 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:39.042612 (+ 1193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:39.044290 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:39.044296 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:39.044301 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:39.044517 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:39.044777 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:39.044779 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:39.044780 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:39.044886 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:39.045228 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:39.045230 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:39.045303 (+ 73us) client_negotiation.cc:769] Sending connection context
0926 16:42:39.045381 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 16:42:39.045478 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181}
I20260926 16:42:39.045663 28379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.037899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57992 (local address 127.21.12.125:35885)
0926 16:42:39.038159 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:39.038170 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:39.038185 (+ 15us) server_negotiation.cc:407] Connection header received
0926 16:42:39.038227 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:39.038230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:39.038276 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:39.038364 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:39.038715 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.039756 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.040483 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.040714 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.040803 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.042743 (+ 1940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:39.042746 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:39.042748 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:39.042770 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:39.044151 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:39.044605 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:39.044607 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:39.044609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:39.044683 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:39.044992 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:39.044994 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:39.044995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:39.045151 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:39.045225 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:39.045438 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 16:42:39.045547 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":78,"threads_started":1}
I20260926 16:42:39.046603 28375 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.124" port: 42743 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:42:39 dist-test-slave-n1hw krb5kdc[27611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440958, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.124@KRBTEST.COM for oryx/127.21.12.124@KRBTEST.COM
I20260926 16:42:39.054200 28377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.047044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:42743 (local address 127.0.0.1:48070)
0926 16:42:39.047157 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:39.047188 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:39.047267 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:39.047471 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:39.047474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:39.047623 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:39.047679 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:39.047683 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.048385 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.048388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:39.048869 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:39.048876 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.049308 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.049311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:39.049410 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.049892 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:39.049912 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:39.051116 (+ 1204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:39.052766 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:39.052769 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:39.052770 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:39.053016 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:39.053327 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:39.053329 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:39.053330 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:39.053409 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:39.053816 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:39.053819 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:39.053900 (+ 81us) client_negotiation.cc:769] Sending connection context
0926 16:42:39.053991 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 16:42:39.054082 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":6}
I20260926 16:42:39.054265 28378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.047142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48070 (local address 127.21.12.124:42743)
0926 16:42:39.047250 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:39.047254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:39.047267 (+ 13us) server_negotiation.cc:407] Connection header received
0926 16:42:39.047340 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:39.047345 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:39.047384 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:39.047462 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:39.047782 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.048265 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.048976 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.049207 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.049311 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.051245 (+ 1934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:39.051248 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:39.051250 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:39.051270 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:39.052664 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:39.053130 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:39.053133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:39.053134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:39.053219 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:39.053521 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:39.053523 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:39.053524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:39.053675 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:39.053822 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:39.054043 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 16:42:39.054134 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":48}
I20260926 16:42:39.057442 28375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be: Bootstrap starting.
I20260926 16:42:39.059753 28375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:39.060034 27847 heartbeater.cc:503] Master 127.21.12.126:34799 was elected leader, sending a full tablet report...
I20260926 16:42:39.060545 28375 log.cc:824] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:39.062637 28375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be: No bootstrap required, opened a new log
I20260926 16:42:39.065842 28375 raft_consensus.cc:348] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } peers { permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 35885 } } peers { permanent_uuid: "324ff229b53c4d59b8116721df0529be" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 42743 } }
I20260926 16:42:39.066073 28375 raft_consensus.cc:374] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:39.066150 28375 raft_consensus.cc:729] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 324ff229b53c4d59b8116721df0529be, State: Initialized, Role: FOLLOWER
I20260926 16:42:39.066581 28375 consensus_queue.cc:260] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } peers { permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 35885 } } peers { permanent_uuid: "324ff229b53c4d59b8116721df0529be" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 42743 } }
I20260926 16:42:39.067036 28381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } peers { permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 35885 } } peers { permanent_uuid: "324ff229b53c4d59b8116721df0529be" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 42743 } } }
I20260926 16:42:39.067147 28381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:42:39.067740 28375 sys_catalog.cc:564] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [sys.catalog]: configured and running, proceeding with master startup.
W20260926 16:42:39.072309 28392 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 16:42:39.072388 28392 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 16:42:39.075657 28221 master_runner.cc:428] Detected that this master 324ff229b53c4d59b8116721df0529be is joining an existing cluster
I20260926 16:42:39.075708 28221 master_runner.cc:432] Initiating AddMaster RPC to add 127.21.12.124:42743
I20260926 16:42:39.086359 28399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.077718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34799 (local address 127.0.0.1:56998)
0926 16:42:39.078112 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:39.078130 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:39.078221 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:39.078535 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:39.078538 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:39.078848 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:39.079001 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:39.079009 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.080107 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.080111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:39.080992 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:39.081003 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.081710 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.081713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:39.081826 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.082319 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:39.082340 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:39.082771 (+ 431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:39.084878 (+ 2107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:39.084880 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:39.084882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:39.085132 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:39.085405 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:39.085407 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:39.085409 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:39.085541 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:39.085959 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:39.085962 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:39.086038 (+ 76us) client_negotiation.cc:769] Sending connection context
0926 16:42:39.086121 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 16:42:39.086221 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":110,"threads_started":1}
I20260926 16:42:39.086603 27695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.078176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56998 (local address 127.21.12.126:34799)
0926 16:42:39.078303 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:39.078308 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:39.078322 (+ 14us) server_negotiation.cc:407] Connection header received
0926 16:42:39.078366 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:39.078369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:39.078415 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:39.078496 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:39.079147 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.079967 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.081167 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.081551 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.081662 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.082941 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:39.082946 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:39.082949 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:39.082975 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:39.084732 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:39.085224 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:39.085228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:39.085230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:39.085316 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:39.085632 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:39.085635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:39.085640 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:39.085870 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:39.085971 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:39.086194 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 16:42:39.086305 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 16:42:39.088689 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 27718
I20260926 16:42:39.095860 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.65:37069
--local_ip_for_outbound_sockets=127.21.12.65
--tserver_master_addrs=127.21.12.126:34799
--webserver_port=39041
--webserver_interface=127.21.12.65
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.65.keytab
--principal=oryx/127.21.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:38299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.12.126:34799,127.21.12.125:35885
--tserver_master_addrs=127.21.12.126:34799,127.21.12.125:35885,127.21.12.124:42743 with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 16:42:39 dist-test-slave-n1hw krb5kdc[27611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440957, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.126@KRBTEST.COM for oryx/127.21.12.124@KRBTEST.COM
I20260926 16:42:39.102005 28398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.077801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.125:35885 (local address 127.0.0.1:57998)
0926 16:42:39.078250 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:39.078263 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:39.078363 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:39.078697 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:39.078703 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:39.078946 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:39.079096 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:39.079104 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.079909 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.079912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:39.080504 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:39.080512 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.080980 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.080983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:39.081086 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.081662 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:39.081676 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:39.098045 (+ 16369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:39.100021 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:39.100023 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:39.100026 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:39.100265 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:39.100639 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:39.100642 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:39.100644 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:39.100741 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:39.101204 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:39.101207 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:39.101295 (+ 88us) client_negotiation.cc:769] Sending connection context
0926 16:42:39.101392 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 16:42:39.101590 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":104,"threads_started":1}
I20260926 16:42:39.102051 28379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.077860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57998 (local address 127.21.12.125:35885)
0926 16:42:39.078118 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:39.078123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:39.078336 (+ 213us) server_negotiation.cc:407] Connection header received
0926 16:42:39.078451 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:39.078457 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:39.078508 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:39.078572 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:39.079261 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.079786 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.080632 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.080871 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.080979 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.098217 (+ 17238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:39.098221 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:39.098224 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:39.098256 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:39.099892 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:39.100436 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:39.100440 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:39.100442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:39.100537 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:39.100843 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:39.100846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:39.100847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:39.101068 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:39.101162 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:39.101742 (+ 580us) server_negotiation.cc:299] Negotiation successful
0926 16:42:39.101894 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260926 16:42:39.104744 28213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.094618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:42743 (local address 127.0.0.1:48088)
0926 16:42:39.094934 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:39.094950 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:39.095037 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:39.095318 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:39.095323 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:39.095530 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:39.095688 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:39.095695 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.096513 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.096520 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:39.097060 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:39.097068 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.097800 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.097805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:39.097908 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.098424 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:39.098440 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:39.100472 (+ 2032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:39.102875 (+ 2403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:39.102878 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:39.102880 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:39.103132 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:39.103472 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:39.103478 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:39.103480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:39.103846 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:39.104343 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:39.104346 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:39.104443 (+ 97us) client_negotiation.cc:769] Sending connection context
0926 16:42:39.104514 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 16:42:39.104625 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221}
I20260926 16:42:39.105424 28401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.094683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48088 (local address 127.21.12.124:42743)
0926 16:42:39.094976 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:39.094983 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:39.094999 (+ 16us) server_negotiation.cc:407] Connection header received
0926 16:42:39.095160 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:39.095163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:39.095215 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:39.095302 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:39.095810 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.096397 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.097233 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.097635 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.097777 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.100611 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:39.100614 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:39.100617 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:39.100639 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:39.102751 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:39.103254 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:39.103260 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:39.103262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:39.103345 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:39.103971 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:39.103977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:39.103979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:39.104227 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:39.104968 (+ 741us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:39.105066 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 16:42:39.105223 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":97,"threads_started":1}
I20260926 16:42:39.106359 27656 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.21.12.124:42743: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "324ff229b53c4d59b8116721df0529be"
member_type: NON_VOTER
last_known_addr {
host: "127.21.12.124"
port: 42743
}
attrs {
promote: true
}
}
dest_uuid: "39e91c6cfaa7419891443717312b224d"
cas_config_opid_index: 5
I20260926 16:42:39.106904 28378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.078803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48086 (local address 127.21.12.124:42743)
0926 16:42:39.078906 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:39.078910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:39.081350 (+ 2440us) server_negotiation.cc:407] Connection header received
0926 16:42:39.081523 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:39.081527 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:39.081571 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:39.081613 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:39.098575 (+ 16962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.099363 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.100498 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.100825 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.100934 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.103324 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:39.103327 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:39.103330 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:39.103353 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:39.104990 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:39.105434 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:39.105437 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:39.105439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:39.105526 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:39.105966 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:39.105970 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:39.105971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:39.106162 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:39.106252 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:39.106569 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 16:42:39.106769 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":79,"server-negotiator.queue_time_us":41}
I20260926 16:42:39.107193 28400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.078969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:42743 (local address 127.0.0.1:48086)
0926 16:42:39.081275 (+ 2306us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:39.081288 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:39.081404 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:39.082060 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:39.082062 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:39.098303 (+ 16241us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:39.098447 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:39.098455 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.099492 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.099495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:39.100368 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:39.100381 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.101057 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.101064 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:39.101182 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.101613 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:39.101632 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:39.103201 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:39.105087 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:39.105089 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:39.105091 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:39.105336 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:39.105607 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:39.105610 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:39.105612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:39.105704 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:39.106244 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:39.106249 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:39.106355 (+ 106us) client_negotiation.cc:769] Sending connection context
0926 16:42:39.106479 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 16:42:39.107058 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2247,"thread_start_us":98,"threads_started":1}
I20260926 16:42:39.107179 27656 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } peers { permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: NON_VOTER last_known_addr { host: "127.21.12.125" port: 35885 } attrs { promote: true } } peers { permanent_uuid: "324ff229b53c4d59b8116721df0529be" member_type: NON_VOTER last_known_addr { host: "127.21.12.124" port: 42743 } attrs { promote: true } }
I20260926 16:42:39.108421 28217 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 324ff229b53c4d59b8116721df0529be (127.21.12.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } peers { permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: NON_VOTER last_known_addr { host: "127.21.12.125" port: 35885 } attrs { promote: true } } peers { permanent_uuid: "324ff229b53c4d59b8116721df0529be" member_type: NON_VOTER last_known_addr { host: "127.21.12.124" port: 42743 } attrs { promote: true } } }
I20260926 16:42:39.109946 28217 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.21.12.124:42743
I20260926 16:42:39.110140 28217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "39e91c6cfaa7419891443717312b224d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } peers { permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: NON_VOTER last_known_addr { host: "127.21.12.125" port: 35885 } attrs { promote: true } } peers { permanent_uuid: "324ff229b53c4d59b8116721df0529be" member_type: NON_VOTER last_known_addr { host: "127.21.12.124" port: 42743 } attrs { promote: true } } }
I20260926 16:42:39.110241 28217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:39.111011 28349 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:39.112403 28213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.108274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.125:35885 (local address 127.0.0.1:58014)
0926 16:42:39.108377 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:39.108392 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:39.108476 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:39.108923 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:39.108928 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:39.108950 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:39.109063 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:39.109073 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.110180 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.110183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:39.111314 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:39.111324 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.112031 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.112034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:39.112115 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.112154 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 16:42:39.112220 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 16:42:39.112285 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260926 16:42:39.112458 28379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.108360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58014 (local address 127.21.12.125:35885)
0926 16:42:39.108505 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:39.108511 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:39.108529 (+ 18us) server_negotiation.cc:407] Connection header received
0926 16:42:39.108582 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:39.108588 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:39.108666 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:39.108777 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:39.109208 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.110046 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.111438 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.111904 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.112030 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.112201 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:39.112236 (+ 35us) server_negotiation.cc:299] Negotiation successful
0926 16:42:39.112302 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 16:42:39.113132 28349 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [term 1 FOLLOWER]: Refusing update from remote peer 39e91c6cfaa7419891443717312b224d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 16:42:39.113476 28221 master.cc:537] Master@127.21.12.124:42743 shutting down...
I20260926 16:42:39.114400 28259 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [term 1 LEARNER]: Refusing update from remote peer 39e91c6cfaa7419891443717312b224d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 16:42:39.114872 28218 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: NON_VOTER last_known_addr { host: "127.21.12.125" port: 35885 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 16:42:39.115345 28218 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d [LEADER]: Connected to new peer: Peer: permanent_uuid: "324ff229b53c4d59b8116721df0529be" member_type: NON_VOTER last_known_addr { host: "127.21.12.124" port: 42743 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:42:39.116916 28259 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 324ff229b53c4d59b8116721df0529be (127.21.12.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } peers { permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: NON_VOTER last_known_addr { host: "127.21.12.125" port: 35885 } attrs { promote: true } } peers { permanent_uuid: "324ff229b53c4d59b8116721df0529be" member_type: NON_VOTER last_known_addr { host: "127.21.12.124" port: 42743 } attrs { promote: true } } }
W20260926 16:42:39.117249 27642 consensus_peers.cc:487] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d -> Peer 324ff229b53c4d59b8116721df0529be (127.21.12.124:42743): Couldn't send request to peer 324ff229b53c4d59b8116721df0529be. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 16:42:39.118695 28296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "39e91c6cfaa7419891443717312b224d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } peers { permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: NON_VOTER last_known_addr { host: "127.21.12.125" port: 35885 } attrs { promote: true } } peers { permanent_uuid: "324ff229b53c4d59b8116721df0529be" member_type: NON_VOTER last_known_addr { host: "127.21.12.124" port: 42743 } attrs { promote: true } } }
I20260926 16:42:39.118814 28296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [sys.catalog]: This master's current role is: LEARNER
I20260926 16:42:39.118734 28404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39e91c6cfaa7419891443717312b224d. Latest consensus state: current_term: 1 leader_uuid: "39e91c6cfaa7419891443717312b224d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } peers { permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: NON_VOTER last_known_addr { host: "127.21.12.125" port: 35885 } attrs { promote: true } } peers { permanent_uuid: "324ff229b53c4d59b8116721df0529be" member_type: NON_VOTER last_known_addr { host: "127.21.12.124" port: 42743 } attrs { promote: true } } }
I20260926 16:42:39.118870 28404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [sys.catalog]: This master's current role is: LEARNER
I20260926 16:42:39.120101 28221 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:42:39.120368 28221 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:42:39.120443 28218 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d: attempting to promote NON_VOTER 1f70fdb997ce4656b970b08582ad2556 to VOTER
I20260926 16:42:39.120477 28221 tablet_replica.cc:330] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be: stopping tablet replica
I20260926 16:42:39.120865 28218 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } peers { permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 35885 } attrs { promote: false } } peers { permanent_uuid: "324ff229b53c4d59b8116721df0529be" member_type: NON_VOTER last_known_addr { host: "127.21.12.124" port: 42743 } attrs { promote: true } }
I20260926 16:42:39.122267 28259 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [term 1 LEARNER]: Refusing update from remote peer 39e91c6cfaa7419891443717312b224d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260926 16:42:39.122421 27642 consensus_peers.cc:487] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d -> Peer 324ff229b53c4d59b8116721df0529be (127.21.12.124:42743): Couldn't send request to peer 324ff229b53c4d59b8116721df0529be. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 16:42:39.122622 28216 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 35885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 16:42:39.124068 28217 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1f70fdb997ce4656b970b08582ad2556 (127.21.12.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } peers { permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 35885 } attrs { promote: false } } peers { permanent_uuid: "324ff229b53c4d59b8116721df0529be" member_type: NON_VOTER last_known_addr { host: "127.21.12.124" port: 42743 } attrs { promote: true } } }
I20260926 16:42:39.124578 28259 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1f70fdb997ce4656b970b08582ad2556 (127.21.12.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } peers { permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 35885 } attrs { promote: false } } peers { permanent_uuid: "324ff229b53c4d59b8116721df0529be" member_type: NON_VOTER last_known_addr { host: "127.21.12.124" port: 42743 } attrs { promote: true } } }
I20260926 16:42:39.125494 28217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "39e91c6cfaa7419891443717312b224d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } peers { permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 35885 } attrs { promote: false } } peers { permanent_uuid: "324ff229b53c4d59b8116721df0529be" member_type: NON_VOTER last_known_addr { host: "127.21.12.124" port: 42743 } attrs { promote: true } } }
I20260926 16:42:39.125612 28217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:39.125659 28216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "39e91c6cfaa7419891443717312b224d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } peers { permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 35885 } attrs { promote: false } } peers { permanent_uuid: "324ff229b53c4d59b8116721df0529be" member_type: NON_VOTER last_known_addr { host: "127.21.12.124" port: 42743 } attrs { promote: true } } }
I20260926 16:42:39.125778 28216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:39.126246 28404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "39e91c6cfaa7419891443717312b224d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } peers { permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 35885 } attrs { promote: false } } peers { permanent_uuid: "324ff229b53c4d59b8116721df0529be" member_type: NON_VOTER last_known_addr { host: "127.21.12.124" port: 42743 } attrs { promote: true } } }
I20260926 16:42:39.126360 28404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:42:39.128368 28221 master.cc:552] Master@127.21.12.124:42743 shutdown complete.
I20260926 16:42:39.128449 28221 master_runner.cc:305] Clearing existing system tablet
I20260926 16:42:39.128727 28221 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:39.128811 28221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:39.130165 28221 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 16:42:39.131155 28406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:39.131384 28221 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:39.131644 28221 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 16:42:39.131721 28221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-2/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-2/wal
uuid: "324ff229b53c4d59b8116721df0529be"
format_stamp: "Formatted at 2026-09-26 16:42:38 on dist-test-slave-n1hw"
server_key: "937bd32e3c38cfad84c86c0d0bebf3eb"
server_key_iv: "fded54cb6ebf68b121d50a99af963889"
server_key_version: "encryptionkey@0"
I20260926 16:42:39.133951 28221 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:42:39.136300 28221 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 16:42:39.136451 28221 log.cc:1196] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 16:42:39.136763 28221 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be: Deleting consensus metadata
I20260926 16:42:39.136969 28221 master_runner.cc:315] Copying system tablet from 127.21.12.126:34799
I20260926 16:42:39.138154 28221 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be: tablet copy: Beginning tablet copy session from remote peer at address 127.21.12.126:34799
I20260926 16:42:39.151700 27695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.138831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57012 (local address 127.21.12.126:34799)
0926 16:42:39.138946 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:39.138952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:39.143070 (+ 4118us) server_negotiation.cc:407] Connection header received
0926 16:42:39.143288 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:39.143292 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:39.143351 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:39.143443 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:39.144031 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.144892 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.146065 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.146395 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.146503 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.147822 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:39.147826 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:39.147829 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:39.147857 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:39.149728 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:39.150303 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:39.150307 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:39.150309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:39.150399 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:39.150742 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:39.150745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:39.150747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:39.150960 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:39.151059 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:39.151419 (+ 360us) server_negotiation.cc:299] Negotiation successful
0926 16:42:39.151537 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 16:42:39.151741 28412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.138737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34799 (local address 127.0.0.1:57012)
0926 16:42:39.142961 (+ 4224us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:39.142977 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:39.143157 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:39.143476 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:39.143479 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:39.143719 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:39.143894 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:39.143902 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.145010 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.145013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:39.145939 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:39.145950 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.146523 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.146526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:39.146623 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.147227 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:39.147241 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:39.147673 (+ 432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:39.149889 (+ 2216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:39.149893 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:39.149896 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:39.150159 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:39.150504 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:39.150509 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:39.150513 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:39.150633 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:39.151068 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:39.151071 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:39.151193 (+ 122us) client_negotiation.cc:769] Sending connection context
0926 16:42:39.151302 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 16:42:39.151535 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4146,"thread_start_us":84,"threads_started":1}
I20260926 16:42:39.152726 27676 tablet_copy_service.cc:140] P 39e91c6cfaa7419891443717312b224d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 324ff229b53c4d59b8116721df0529be ({username='oryx', principal='oryx/127.21.12.124@KRBTEST.COM'} at 127.0.0.1:57012)
I20260926 16:42:39.152828 27676 tablet_copy_service.cc:161] P 39e91c6cfaa7419891443717312b224d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 324ff229b53c4d59b8116721df0529be at {username='oryx', principal='oryx/127.21.12.124@KRBTEST.COM'} at 127.0.0.1:57012: session id = 324ff229b53c4d59b8116721df0529be-00000000000000000000000000000000
I20260926 16:42:39.153926 27676 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 16:42:39.155150 28221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 16:42:39.157522 28221 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be: tablet copy: Starting download of 0 data blocks...
I20260926 16:42:39.157689 28221 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be: tablet copy: Starting download of 1 WAL segments...
I20260926 16:42:39.159427 28221 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 16:42:39.161139 27676 tablet_copy_service.cc:342] P 39e91c6cfaa7419891443717312b224d: Request end of tablet copy session 324ff229b53c4d59b8116721df0529be-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.21.12.124@KRBTEST.COM'} at 127.0.0.1:57012
I20260926 16:42:39.161218 27676 tablet_copy_service.cc:433] P 39e91c6cfaa7419891443717312b224d: ending tablet copy session 324ff229b53c4d59b8116721df0529be-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 324ff229b53c4d59b8116721df0529be
I20260926 16:42:39.168565 28221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:39.170748 28414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:39.170984 28416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:39.171185 28413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:39.171362 28221 server_base.cc:1034] running on GCE node
I20260926 16:42:39.171619 28221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:39.171898 28221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:39.173029 28221 hybrid_clock.cc:648] HybridClock initialized: now 1790440959173011 us; error 33 us; skew 500 ppm
Sep 26 16:42:39 dist-test-slave-n1hw krb5kdc[27611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440959, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:39.174830 28221 init.cc:377] Logged in from keytab as oryx/127.21.12.124@KRBTEST.COM (short username oryx)
I20260926 16:42:39.175344 28221 webserver.cc:460] Webserver started at http://127.21.12.124:35265/ using document root <none> and password file <none>
I20260926 16:42:39.175608 28221 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:39.175786 28221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:39.177194 28221 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 16:42:39.178270 28423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:39.178612 28221 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 16:42:39.178723 28221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-2/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-2/wal
uuid: "324ff229b53c4d59b8116721df0529be"
format_stamp: "Formatted at 2026-09-26 16:42:38 on dist-test-slave-n1hw"
server_key: "937bd32e3c38cfad84c86c0d0bebf3eb"
server_key_iv: "fded54cb6ebf68b121d50a99af963889"
server_key_version: "encryptionkey@0"
I20260926 16:42:39.178853 28221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:39.197083 28221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:39.199469 28221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:39.204450 28221 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.124:42743
I20260926 16:42:39.204478 28475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.124:42743 every 8 connection(s)
I20260926 16:42:39.205155 28221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/master-2/data/info.pb
I20260926 16:42:39.207192 28476 sys_catalog.cc:263] Verifying existing consensus state
I20260926 16:42:39.208498 28476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be: Bootstrap starting.
I20260926 16:42:39.220480 28476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 16:42:39.220819 28476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be: Bootstrap complete.
I20260926 16:42:39.221468 28476 raft_consensus.cc:348] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } peers { permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 35885 } attrs { promote: false } } peers { permanent_uuid: "324ff229b53c4d59b8116721df0529be" member_type: NON_VOTER last_known_addr { host: "127.21.12.124" port: 42743 } attrs { promote: true } }
I20260926 16:42:39.221622 28476 raft_consensus.cc:729] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 324ff229b53c4d59b8116721df0529be, State: Initialized, Role: LEARNER
I20260926 16:42:39.221769 28476 consensus_queue.cc:260] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } peers { permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 35885 } attrs { promote: false } } peers { permanent_uuid: "324ff229b53c4d59b8116721df0529be" member_type: NON_VOTER last_known_addr { host: "127.21.12.124" port: 42743 } attrs { promote: true } }
I20260926 16:42:39.222126 28479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } peers { permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 35885 } attrs { promote: false } } peers { permanent_uuid: "324ff229b53c4d59b8116721df0529be" member_type: NON_VOTER last_known_addr { host: "127.21.12.124" port: 42743 } attrs { promote: true } } }
I20260926 16:42:39.222271 28476 sys_catalog.cc:564] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:42:39.222303 28479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [sys.catalog]: This master's current role is: LEARNER
I20260926 16:42:39.226187 28490 catalog_manager.cc:1259] Loaded cluster ID: c302d5e480d84aabadda95d4bba72d12
I20260926 16:42:39.226249 28490 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be: loading cluster ID for follower catalog manager: success
I20260926 16:42:39.228358 28490 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be: acquiring CA information for follower catalog manager: success
I20260926 16:42:39.229732 28490 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 16:42:39.248672 28402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:39.248900 28402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:39.248942 28402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:39.253378 28402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:39.253454 28402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:39.253659 28402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.65
I20260926 16:42:39.259512 28402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:38299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.65.keytab
--principal=oryx/127.21.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.65:37069
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.65
--webserver_port=39041
--tserver_master_addrs=127.21.12.126:34799,127.21.12.125:35885,127.21.12.124:42743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.65
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:39.260520 28402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:39.261499 28402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:39.270954 28495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:39.270987 28498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:39.270987 28496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:39.271049 28402 server_base.cc:1034] running on GCE node
I20260926 16:42:39.271623 28402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:39.272162 28402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:39.273330 28402 hybrid_clock.cc:648] HybridClock initialized: now 1790440959273297 us; error 44 us; skew 500 ppm
I20260926 16:42:39.274590 27981 heartbeater.cc:503] Master 127.21.12.126:34799 was elected leader, sending a full tablet report...
Sep 26 16:42:39 dist-test-slave-n1hw krb5kdc[27611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440959, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:39.276351 28402 init.cc:377] Logged in from keytab as oryx/127.21.12.65@KRBTEST.COM (short username oryx)
I20260926 16:42:39.277146 28402 webserver.cc:460] Webserver started at http://127.21.12.65:39041/ using document root <none> and password file <none>
I20260926 16:42:39.277631 28402 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:39.277697 28402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:39.281515 28402 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:42:39.283565 28505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:39.284904 28402 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 16:42:39.285022 28402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-0/wal
uuid: "2d57671a949840679f2e9b9ec4ddb8de"
format_stamp: "Formatted at 2026-09-26 16:42:37 on dist-test-slave-n1hw"
server_key: "24add1f576cd6a08a45428d5a0557c98"
server_key_iv: "0b041479eeeb36a4d0f29a7dda71dd3f"
server_key_version: "encryptionkey@0"
I20260926 16:42:39.285576 28402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:39.307777 28402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:39.311183 28402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:39.311336 28402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:39.311883 28402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:42:39.312850 28402 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:39.312898 28402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:39.312965 28402 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:39.312992 28402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:39.322819 28402 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.65:37069
I20260926 16:42:39.322880 28618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.65:37069 every 8 connection(s)
I20260926 16:42:39.323931 28402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
I20260926 16:42:39.330930 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 28402
I20260926 16:42:39.331120 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 27852
Sep 26 16:42:39 dist-test-slave-n1hw krb5kdc[27611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440959, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.65@KRBTEST.COM for oryx/127.21.12.124@KRBTEST.COM
Sep 26 16:42:39 dist-test-slave-n1hw krb5kdc[27611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440959, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.65@KRBTEST.COM for oryx/127.21.12.125@KRBTEST.COM
Sep 26 16:42:39 dist-test-slave-n1hw krb5kdc[27611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440959, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.65@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM
I20260926 16:42:39.341382 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.66:41203
--local_ip_for_outbound_sockets=127.21.12.66
--tserver_master_addrs=127.21.12.126:34799
--webserver_port=36899
--webserver_interface=127.21.12.66
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.66.keytab
--principal=oryx/127.21.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:38299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.12.126:34799,127.21.12.125:35885
--tserver_master_addrs=127.21.12.126:34799,127.21.12.125:35885,127.21.12.124:42743 with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:42:39.342525 28379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.326438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:56291 (local address 127.21.12.125:35885)
0926 16:42:39.326580 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:39.326585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:39.328419 (+ 1834us) server_negotiation.cc:407] Connection header received
0926 16:42:39.328627 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:39.328630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:39.328693 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:39.328778 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:39.329799 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.330603 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.332639 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.332986 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.333062 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.337442 (+ 4380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:39.337445 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:39.337451 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:39.337478 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:39.339489 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:39.340571 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:39.340579 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:39.340584 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:39.340693 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:39.341125 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:39.341133 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:39.341140 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:39.341419 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:39.341518 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:39.342163 (+ 645us) server_negotiation.cc:299] Negotiation successful
0926 16:42:39.342325 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 16:42:39.342559 28623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.326129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:47217 (local address 127.21.12.124:42743)
0926 16:42:39.326393 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:39.326397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:39.326966 (+ 569us) server_negotiation.cc:407] Connection header received
0926 16:42:39.327944 (+ 978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:39.327947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:39.328000 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:39.328083 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:39.329778 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.330485 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.332671 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.333136 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.333229 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.337860 (+ 4631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:39.337867 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:39.337873 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:39.337907 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:39.339941 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:39.340483 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:39.340489 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:39.340494 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:39.340591 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:39.341020 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:39.341024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:39.341026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:39.341254 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:39.341333 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:39.342143 (+ 810us) server_negotiation.cc:299] Negotiation successful
0926 16:42:39.342294 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":82,"threads_started":1}
I20260926 16:42:39.344517 28624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.326244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:42743 (local address 127.21.12.65:47217)
0926 16:42:39.326823 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:39.326874 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:39.327719 (+ 845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:39.328290 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:39.328299 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:39.329000 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:39.329614 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:39.329629 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.330634 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.330637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:39.332520 (+ 1883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:39.332539 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.333411 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.333417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:39.333639 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.334442 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:39.334478 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:39.337672 (+ 3194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:39.340077 (+ 2405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:39.340083 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:39.340087 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:39.340366 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:39.340708 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:39.340714 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:39.340718 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:39.340892 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:39.341422 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:39.341429 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:39.341802 (+ 373us) client_negotiation.cc:769] Sending connection context
0926 16:42:39.342725 (+ 923us) client_negotiation.cc:240] Negotiation successful
0926 16:42:39.343356 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":434,"spinlock_wait_cycles":32768,"thread_start_us":674,"threads_started":1}
I20260926 16:42:39.344528 28626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.326244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.125:35885 (local address 127.21.12.65:56291)
0926 16:42:39.328341 (+ 2097us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:39.328354 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:39.328484 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:39.328899 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:39.328905 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:39.329103 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:39.329628 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:39.329637 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.330766 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.330770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:39.332495 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:39.332506 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.333259 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.333263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:39.333621 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.334609 (+ 988us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:39.334626 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:39.337190 (+ 2564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:39.339646 (+ 2456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:39.339653 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:39.339670 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:39.340097 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:39.340864 (+ 767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:39.340868 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:39.340872 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:39.340982 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:39.341611 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:39.341614 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:39.341802 (+ 188us) client_negotiation.cc:769] Sending connection context
0926 16:42:39.342782 (+ 980us) client_negotiation.cc:240] Negotiation successful
0926 16:42:39.343209 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2000,"mutex_wait_us":1054,"spinlock_wait_cycles":35968,"thread_start_us":121,"threads_started":1}
I20260926 16:42:39.345640 28625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.326244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34799 (local address 127.21.12.65:49557)
0926 16:42:39.328022 (+ 1778us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:39.328032 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:39.328143 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:39.328379 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:39.328383 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:39.329005 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:39.329629 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:39.329637 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.330766 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.330770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:39.332503 (+ 1733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:39.332508 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.333259 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.333263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:39.333587 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.334483 (+ 896us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:39.334499 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:39.338661 (+ 4162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:39.342473 (+ 3812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:39.342476 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:39.342478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:39.343228 (+ 750us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:39.343642 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:39.343645 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:39.343649 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:39.343770 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:39.344438 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:39.344441 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:39.344546 (+ 105us) client_negotiation.cc:769] Sending connection context
0926 16:42:39.345231 (+ 685us) client_negotiation.cc:240] Negotiation successful
0926 16:42:39.345432 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1693,"mutex_wait_us":331,"spinlock_wait_cycles":63104,"thread_start_us":143,"threads_started":1}
I20260926 16:42:39.346046 28619 heartbeater.cc:347] Connected to a master server at 127.21.12.124:42743
I20260926 16:42:39.346091 28621 heartbeater.cc:347] Connected to a master server at 127.21.12.125:35885
I20260926 16:42:39.346287 28619 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:39.346287 28621 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:39.346431 27695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.325728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:49557 (local address 127.21.12.126:34799)
0926 16:42:39.325907 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:39.325911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:39.328152 (+ 2241us) server_negotiation.cc:407] Connection header received
0926 16:42:39.328212 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:39.328215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:39.328261 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:39.328362 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:39.329796 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.330601 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.332640 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.332988 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.333062 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.339823 (+ 6761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:39.339827 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:39.339831 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:39.339865 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:39.342050 (+ 2185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:39.343382 (+ 1332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:39.343387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:39.343389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:39.343496 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:39.343914 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:39.343919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:39.343920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:39.344237 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:39.344349 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:39.346162 (+ 1813us) server_negotiation.cc:299] Negotiation successful
0926 16:42:39.346281 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260926 16:42:39.346844 28621 heartbeater.cc:511] Master 127.21.12.125:35885 requested a full tablet report, sending...
I20260926 16:42:39.346843 28619 heartbeater.cc:511] Master 127.21.12.124:42743 requested a full tablet report, sending...
I20260926 16:42:39.347452 28620 heartbeater.cc:347] Connected to a master server at 127.21.12.126:34799
I20260926 16:42:39.347537 28620 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:39.347725 28620 heartbeater.cc:511] Master 127.21.12.126:34799 requested a full tablet report, sending...
I20260926 16:42:39.348534 27656 ts_manager.cc:194] Re-registered known tserver with Master: 2d57671a949840679f2e9b9ec4ddb8de (127.21.12.65:37069)
I20260926 16:42:39.348608 28255 ts_manager.cc:194] Registered new tserver with Master: 2d57671a949840679f2e9b9ec4ddb8de (127.21.12.65:37069)
I20260926 16:42:39.348606 28440 ts_manager.cc:194] Registered new tserver with Master: 2d57671a949840679f2e9b9ec4ddb8de (127.21.12.65:37069)
I20260926 16:42:39.349416 27656 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.12.65@KRBTEST.COM'} at 127.21.12.65:49557
W20260926 16:42:39.454972 28627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:39.455209 28627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:39.455251 28627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:39.459187 28627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:39.459233 28627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:39.459290 28627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.66
I20260926 16:42:39.464237 28627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:38299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.66.keytab
--principal=oryx/127.21.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.66:41203
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.66
--webserver_port=36899
--tserver_master_addrs=127.21.12.126:34799,127.21.12.125:35885,127.21.12.124:42743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.66
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:39.465150 28627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:39.466104 28627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:39.475759 28635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:39.475780 28632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:39.475869 28633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:39.476257 28627 server_base.cc:1034] running on GCE node
I20260926 16:42:39.476612 28627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:39.477257 28627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:39.478442 28627 hybrid_clock.cc:648] HybridClock initialized: now 1790440959478407 us; error 48 us; skew 500 ppm
Sep 26 16:42:39 dist-test-slave-n1hw krb5kdc[27611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440959, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:39.481185 28627 init.cc:377] Logged in from keytab as oryx/127.21.12.66@KRBTEST.COM (short username oryx)
I20260926 16:42:39.482199 28627 webserver.cc:460] Webserver started at http://127.21.12.66:36899/ using document root <none> and password file <none>
I20260926 16:42:39.482743 28627 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:39.482813 28627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:39.486687 28627 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:42:39.488804 28642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:39.490193 28627 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:42:39.490315 28627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-1/wal
uuid: "0df678e3047b4d9ca7e774b8040f86b4"
format_stamp: "Formatted at 2026-09-26 16:42:38 on dist-test-slave-n1hw"
server_key: "024f1adfbc663ec248a4a15ecce6cf22"
server_key_iv: "c985e37548341c7d524a994afc827bef"
server_key_version: "encryptionkey@0"
I20260926 16:42:39.490844 28627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:39.493687 28115 heartbeater.cc:503] Master 127.21.12.126:34799 was elected leader, sending a full tablet report...
I20260926 16:42:39.530676 28627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:39.533654 28627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:39.533869 28627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:39.534467 28627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:42:39.535287 28627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:39.535355 28627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:39.535426 28627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:39.535477 28627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:39.544420 28627 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.66:41203
I20260926 16:42:39.544521 28755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.66:41203 every 8 connection(s)
I20260926 16:42:39.545425 28627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
I20260926 16:42:39.548635 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 28627
I20260926 16:42:39.548800 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 27986
I20260926 16:42:39.556042 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.67:42229
--local_ip_for_outbound_sockets=127.21.12.67
--tserver_master_addrs=127.21.12.126:34799
--webserver_port=45099
--webserver_interface=127.21.12.67
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.67.keytab
--principal=oryx/127.21.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:38299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.12.126:34799,127.21.12.125:35885
--tserver_master_addrs=127.21.12.126:34799,127.21.12.125:35885,127.21.12.124:42743 with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 16:42:39 dist-test-slave-n1hw krb5kdc[27611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440959, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.66@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM
Sep 26 16:42:39 dist-test-slave-n1hw krb5kdc[27611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440959, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.66@KRBTEST.COM for oryx/127.21.12.124@KRBTEST.COM
Sep 26 16:42:39 dist-test-slave-n1hw krb5kdc[27611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440959, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.66@KRBTEST.COM for oryx/127.21.12.125@KRBTEST.COM
I20260926 16:42:39.565305 27695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.547909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:44569 (local address 127.21.12.126:34799)
0926 16:42:39.548027 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:39.548031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:39.548390 (+ 359us) server_negotiation.cc:407] Connection header received
0926 16:42:39.550465 (+ 2075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:39.550469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:39.550526 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:39.550584 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:39.552660 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.553477 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.555611 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.555945 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.556026 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.560206 (+ 4180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:39.560210 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:39.560213 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:39.560240 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:39.562675 (+ 2435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:39.563400 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:39.563406 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:39.563411 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:39.563519 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:39.563968 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:39.563971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:39.563973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:39.564188 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:39.564270 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:39.565009 (+ 739us) server_negotiation.cc:299] Negotiation successful
0926 16:42:39.565137 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 16:42:39.565305 28623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.547846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:41449 (local address 127.21.12.124:42743)
0926 16:42:39.547978 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:39.547982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:39.548613 (+ 631us) server_negotiation.cc:407] Connection header received
0926 16:42:39.550495 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:39.550498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:39.550552 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:39.550604 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:39.552655 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.553477 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.555692 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.556063 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.556139 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.560206 (+ 4067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:39.560210 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:39.560213 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:39.560240 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:39.562553 (+ 2313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:39.563440 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:39.563447 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:39.563449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:39.563526 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:39.563975 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:39.563979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:39.563980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:39.564188 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:39.564270 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:39.565009 (+ 739us) server_negotiation.cc:299] Negotiation successful
0926 16:42:39.565137 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 16:42:39.565402 28379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.547193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:58293 (local address 127.21.12.125:35885)
0926 16:42:39.547322 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:39.547326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:39.550895 (+ 3569us) server_negotiation.cc:407] Connection header received
0926 16:42:39.551088 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:39.551094 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:39.551154 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:39.551209 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:39.552608 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.553541 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.555609 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.555934 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.556004 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.560264 (+ 4260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:39.560270 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:39.560276 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:39.560312 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:39.562545 (+ 2233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:39.563494 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:39.563499 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:39.563503 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:39.563600 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:39.563995 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:39.564001 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:39.564005 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:39.564269 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:39.564364 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:39.565038 (+ 674us) server_negotiation.cc:299] Negotiation successful
0926 16:42:39.565183 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 16:42:39.567121 28762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.547608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.125:35885 (local address 127.21.12.66:58293)
0926 16:42:39.550797 (+ 3189us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:39.550813 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:39.550970 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:39.551366 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:39.551372 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:39.551687 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:39.552450 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:39.552468 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.553688 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.553694 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:39.555475 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:39.555491 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.556199 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.556205 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:39.556531 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.557569 (+ 1038us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:39.557588 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:39.560091 (+ 2503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:39.562690 (+ 2599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:39.562697 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:39.562714 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:39.563352 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:39.563728 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:39.563734 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:39.563738 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:39.563861 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:39.564430 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:39.564436 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:39.564712 (+ 276us) client_negotiation.cc:769] Sending connection context
0926 16:42:39.565670 (+ 958us) client_negotiation.cc:240] Negotiation successful
0926 16:42:39.566074 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3090,"mutex_wait_us":909,"spinlock_wait_cycles":7936,"thread_start_us":88,"threads_started":1}
I20260926 16:42:39.567384 28761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.547608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:42743 (local address 127.21.12.66:41449)
0926 16:42:39.548521 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:39.548531 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:39.550250 (+ 1719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:39.550992 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:39.551001 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:39.551824 (+ 823us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:39.552446 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:39.552465 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.553663 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.553667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:39.555571 (+ 1904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:39.555582 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.556523 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.556527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:39.556681 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.558054 (+ 1373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:39.558072 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:39.560015 (+ 1943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:39.562690 (+ 2675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:39.562697 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:39.562714 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:39.563331 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:39.563634 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:39.563637 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:39.563639 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:39.563850 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:39.564338 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:39.564344 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:39.564709 (+ 365us) client_negotiation.cc:769] Sending connection context
0926 16:42:39.565532 (+ 823us) client_negotiation.cc:240] Negotiation successful
0926 16:42:39.565962 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"mutex_wait_us":604,"spinlock_wait_cycles":35968,"thread_start_us":141,"threads_started":1}
I20260926 16:42:39.567121 28760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.547607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34799 (local address 127.21.12.66:44569)
0926 16:42:39.548255 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:39.548303 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:39.550222 (+ 1919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:39.550992 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:39.551001 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:39.551688 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:39.552468 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:39.552474 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.553663 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.553667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:39.555469 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:39.555483 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.556200 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.556206 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:39.556527 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.557405 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:39.557425 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:39.560055 (+ 2630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:39.562781 (+ 2726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:39.562786 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:39.562791 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:39.563274 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:39.563634 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:39.563637 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:39.563639 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:39.563866 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:39.564338 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:39.564344 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:39.564709 (+ 365us) client_negotiation.cc:769] Sending connection context
0926 16:42:39.565955 (+ 1246us) client_negotiation.cc:240] Negotiation successful
0926 16:42:39.566147 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"mutex_wait_us":973,"spinlock_wait_cycles":59904,"thread_start_us":153,"threads_started":1}
I20260926 16:42:39.568475 28756 heartbeater.cc:347] Connected to a master server at 127.21.12.124:42743
I20260926 16:42:39.568498 28758 heartbeater.cc:347] Connected to a master server at 127.21.12.125:35885
I20260926 16:42:39.568643 28757 heartbeater.cc:347] Connected to a master server at 127.21.12.126:34799
I20260926 16:42:39.568703 28756 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:39.568703 28757 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:39.568703 28758 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:39.569332 28758 heartbeater.cc:511] Master 127.21.12.125:35885 requested a full tablet report, sending...
I20260926 16:42:39.569336 28757 heartbeater.cc:511] Master 127.21.12.126:34799 requested a full tablet report, sending...
I20260926 16:42:39.569346 28756 heartbeater.cc:511] Master 127.21.12.124:42743 requested a full tablet report, sending...
I20260926 16:42:39.570477 27656 ts_manager.cc:194] Re-registered known tserver with Master: 0df678e3047b4d9ca7e774b8040f86b4 (127.21.12.66:41203)
I20260926 16:42:39.570631 28440 ts_manager.cc:194] Registered new tserver with Master: 0df678e3047b4d9ca7e774b8040f86b4 (127.21.12.66:41203)
I20260926 16:42:39.571218 27656 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.12.66@KRBTEST.COM'} at 127.21.12.66:44569
I20260926 16:42:39.571534 28255 ts_manager.cc:194] Registered new tserver with Master: 0df678e3047b4d9ca7e774b8040f86b4 (127.21.12.66:41203)
W20260926 16:42:39.668023 28763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:39.668244 28763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:39.668284 28763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:39.672416 28763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:39.672467 28763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:39.672529 28763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.67
I20260926 16:42:39.677520 28763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:38299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.67.keytab
--principal=oryx/127.21.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.67:42229
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.67
--webserver_port=45099
--tserver_master_addrs=127.21.12.126:34799,127.21.12.125:35885,127.21.12.124:42743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.67
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:39.678458 28763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:39.679443 28763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:39.689179 28768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:39.689222 28769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:39.689179 28771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:39.689729 28763 server_base.cc:1034] running on GCE node
I20260926 16:42:39.690166 28763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:39.690737 28763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:39.691892 28763 hybrid_clock.cc:648] HybridClock initialized: now 1790440959691872 us; error 27 us; skew 500 ppm
Sep 26 16:42:39 dist-test-slave-n1hw krb5kdc[27611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440959, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:39.694726 28763 init.cc:377] Logged in from keytab as oryx/127.21.12.67@KRBTEST.COM (short username oryx)
I20260926 16:42:39.695641 28763 webserver.cc:460] Webserver started at http://127.21.12.67:45099/ using document root <none> and password file <none>
I20260926 16:42:39.696321 28763 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:39.696453 28763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:39.700241 28763 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260926 16:42:39.702248 28779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:39.703426 28763 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:42:39.703595 28763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-2/wal
uuid: "b27e9cb575dc439eae6edd24bc55de6c"
format_stamp: "Formatted at 2026-09-26 16:42:38 on dist-test-slave-n1hw"
server_key: "88226f8de3bc232d4007ae8c93bd73fa"
server_key_iv: "87d55b0ccf41b0754991b224167cd2e9"
server_key_version: "encryptionkey@0"
I20260926 16:42:39.704105 28763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:39.723248 28763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:39.726253 28763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:39.726442 28763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:39.727005 28763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:42:39.727824 28763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:39.727893 28763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:39.727962 28763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:39.728017 28763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:39.736279 28828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.732708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:42743 (local address 127.0.0.1:54930)
0926 16:42:39.732945 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:39.732960 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:39.733115 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:39.733437 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:39.733440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:39.733460 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:39.733575 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:39.733581 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.734382 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.734386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:39.735172 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:39.735181 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.735863 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.735866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:39.735955 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.735980 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 16:42:39.736068 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 16:42:39.736152 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":60,"threads_started":1}
I20260926 16:42:39.736276 28623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.732758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54930 (local address 127.21.12.124:42743)
0926 16:42:39.732888 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:39.732894 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:39.733063 (+ 169us) server_negotiation.cc:407] Connection header received
0926 16:42:39.733234 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:39.733237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:39.733305 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:39.733422 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:39.733686 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.734270 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.735299 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.735736 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.735857 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.736010 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:39.736081 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 16:42:39.736144 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 16:42:39.737079 28450 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260926 16:42:39.737525 28889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39e91c6cfaa7419891443717312b224d. Latest consensus state: current_term: 1 leader_uuid: "39e91c6cfaa7419891443717312b224d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } peers { permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 35885 } attrs { promote: false } } peers { permanent_uuid: "324ff229b53c4d59b8116721df0529be" member_type: NON_VOTER last_known_addr { host: "127.21.12.124" port: 42743 } attrs { promote: true } } }
I20260926 16:42:39.737830 28889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [sys.catalog]: This master's current role is: LEARNER
I20260926 16:42:39.738343 28763 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.67:42229
I20260926 16:42:39.738363 28894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.67:42229 every 8 connection(s)
I20260926 16:42:39.739341 28763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
I20260926 16:42:39.742580 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 28763
Sep 26 16:42:39 dist-test-slave-n1hw krb5kdc[27611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440959, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.67@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM
Sep 26 16:42:39 dist-test-slave-n1hw krb5kdc[27611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440959, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.67@KRBTEST.COM for oryx/127.21.12.124@KRBTEST.COM
Sep 26 16:42:39 dist-test-slave-n1hw krb5kdc[27611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440959, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.67@KRBTEST.COM for oryx/127.21.12.125@KRBTEST.COM
I20260926 16:42:39.757894 28623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.740971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:57797 (local address 127.21.12.124:42743)
0926 16:42:39.741095 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:39.741099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:39.742104 (+ 1005us) server_negotiation.cc:407] Connection header received
0926 16:42:39.743290 (+ 1186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:39.743294 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:39.743344 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:39.743397 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:39.745354 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.746156 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.748331 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.748650 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.748717 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.752535 (+ 3818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:39.752538 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:39.752540 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:39.752562 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:39.755112 (+ 2550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:39.756122 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:39.756125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:39.756127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:39.756218 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:39.756620 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:39.756624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:39.756625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:39.756829 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:39.756896 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:39.757602 (+ 706us) server_negotiation.cc:299] Negotiation successful
0926 16:42:39.757713 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 16:42:39.758150 28906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.744584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54334 (local address 127.21.12.126:34799)
0926 16:42:39.744812 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:39.744816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:39.745405 (+ 589us) server_negotiation.cc:407] Connection header received
0926 16:42:39.745444 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:39.745446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:39.745501 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:39.745577 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:39.746552 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.747298 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.749181 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.749523 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.749628 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.753645 (+ 4017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:39.753650 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:39.753654 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:39.753683 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:39.756075 (+ 2392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:39.756528 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:39.756530 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:39.756532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:39.756620 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:39.756913 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:39.756916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:39.756918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:39.757146 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:39.757282 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:39.757841 (+ 559us) server_negotiation.cc:299] Negotiation successful
0926 16:42:39.757963 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"spinlock_wait_cycles":1408,"thread_start_us":76,"threads_started":1}
I20260926 16:42:39.758963 27695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.742400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:56201 (local address 127.21.12.126:34799)
0926 16:42:39.742519 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:39.742525 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:39.742686 (+ 161us) server_negotiation.cc:407] Connection header received
0926 16:42:39.743284 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:39.743287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:39.743336 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:39.743389 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:39.745369 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.746172 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.748144 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.748461 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.748557 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.752530 (+ 3973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:39.752534 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:39.752536 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:39.752560 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:39.756020 (+ 3460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:39.756505 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:39.756510 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:39.756512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:39.756611 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:39.756913 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:39.756916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:39.756918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:39.757539 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:39.757629 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:39.758728 (+ 1099us) server_negotiation.cc:299] Negotiation successful
0926 16:42:39.758826 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47,"spinlock_wait_cycles":3456}
I20260926 16:42:39.759471 28899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.741442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:42743 (local address 127.21.12.67:57797)
0926 16:42:39.741963 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:39.742021 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:39.743063 (+ 1042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:39.743764 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:39.743773 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:39.744414 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:39.745203 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:39.745225 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.746332 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.746334 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:39.748205 (+ 1871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:39.748215 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.748886 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.748890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:39.749007 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.750362 (+ 1355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:39.750381 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:39.752349 (+ 1968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:39.755417 (+ 3068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:39.755423 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:39.755439 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:39.756014 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:39.756334 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:39.756337 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:39.756339 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:39.756508 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:39.756974 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:39.756980 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:39.757322 (+ 342us) client_negotiation.cc:769] Sending connection context
0926 16:42:39.758085 (+ 763us) client_negotiation.cc:240] Negotiation successful
0926 16:42:39.758412 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":1116,"spinlock_wait_cycles":17280,"thread_start_us":145,"threads_started":1}
I20260926 16:42:39.759917 28900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.741442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34799 (local address 127.21.12.67:56201)
0926 16:42:39.742594 (+ 1152us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:39.742605 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:39.743070 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:39.743764 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:39.743773 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:39.744407 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:39.745211 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:39.745226 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.746335 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.746338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:39.748020 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:39.748039 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.748709 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.748713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:39.748914 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.749714 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:39.749737 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:39.752358 (+ 2621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:39.756141 (+ 3783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:39.756147 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:39.756151 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:39.756389 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:39.756728 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:39.756731 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:39.756733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:39.756810 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:39.757816 (+ 1006us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:39.757819 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:39.757902 (+ 83us) client_negotiation.cc:769] Sending connection context
0926 16:42:39.759564 (+ 1662us) client_negotiation.cc:240] Negotiation successful
0926 16:42:39.759730 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1079,"mutex_wait_us":611,"spinlock_wait_cycles":53376,"thread_start_us":122,"threads_started":1}
I20260926 16:42:39.760869 28895 heartbeater.cc:347] Connected to a master server at 127.21.12.124:42743
I20260926 16:42:39.760936 28896 heartbeater.cc:347] Connected to a master server at 127.21.12.126:34799
I20260926 16:42:39.761103 28896 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:39.761238 28901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.741657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.125:35885 (local address 127.21.12.67:53081)
0926 16:42:39.742829 (+ 1172us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:39.742839 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:39.743069 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:39.743764 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:39.743773 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:39.744407 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:39.745212 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:39.745226 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.746335 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.746338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:39.748199 (+ 1861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:39.748205 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.748855 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.748859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:39.748932 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.751148 (+ 2216us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:39.751167 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:39.753421 (+ 2254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:39.755522 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:39.755525 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:39.755527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:39.755876 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:39.760075 (+ 4199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:39.760081 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:39.760086 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:39.760181 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:39.760663 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:39.760667 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:39.760767 (+ 100us) client_negotiation.cc:769] Sending connection context
0926 16:42:39.760897 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 16:42:39.761039 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1123,"mutex_wait_us":707,"spinlock_wait_cycles":29568,"thread_start_us":122,"threads_started":1}
I20260926 16:42:39.761667 28896 heartbeater.cc:511] Master 127.21.12.126:34799 requested a full tablet report, sending...
I20260926 16:42:39.761941 28379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.741860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:53081 (local address 127.21.12.125:35885)
0926 16:42:39.741978 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:39.741982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:39.742913 (+ 931us) server_negotiation.cc:407] Connection header received
0926 16:42:39.743290 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:39.743293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:39.743343 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:39.743394 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:39.745374 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.746179 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.748304 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.748635 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.748710 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.753585 (+ 4875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:39.753589 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:39.753591 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:39.753615 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:39.755416 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:39.759833 (+ 4417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:39.759838 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:39.759841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:39.759943 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:39.760287 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:39.760291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:39.760292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:39.760515 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:39.760596 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:39.761663 (+ 1067us) server_negotiation.cc:299] Negotiation successful
0926 16:42:39.761810 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 16:42:39.761103 28895 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:39.762367 28895 heartbeater.cc:511] Master 127.21.12.124:42743 requested a full tablet report, sending...
I20260926 16:42:39.762790 28897 heartbeater.cc:347] Connected to a master server at 127.21.12.125:35885
I20260926 16:42:39.762866 28897 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:39.763013 28897 heartbeater.cc:511] Master 127.21.12.125:35885 requested a full tablet report, sending...
I20260926 16:42:39.763198 28440 ts_manager.cc:194] Registered new tserver with Master: b27e9cb575dc439eae6edd24bc55de6c (127.21.12.67:42229)
I20260926 16:42:39.763058 27656 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54334:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 16:42:39.766834 28910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.749988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40074 (local address 127.21.12.125:35885)
0926 16:42:39.750482 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:39.750498 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:39.750512 (+ 14us) server_negotiation.cc:407] Connection header received
0926 16:42:39.750553 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:39.750557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:39.750595 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:39.750697 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:39.755411 (+ 4714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.756193 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.758275 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.758597 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.758692 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.763357 (+ 4665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:39.763362 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:39.763367 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:39.763394 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:39.765063 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:39.765511 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:39.765514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:39.765516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:39.765609 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:39.765911 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:39.765914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:39.765915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:39.766086 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:39.766245 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:39.766563 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 16:42:39.766683 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":76,"threads_started":1}
I20260926 16:42:39.767387 28255 ts_manager.cc:194] Registered new tserver with Master: b27e9cb575dc439eae6edd24bc55de6c (127.21.12.67:42229)
I20260926 16:42:39.768036 27655 ts_manager.cc:194] Re-registered known tserver with Master: b27e9cb575dc439eae6edd24bc55de6c (127.21.12.67:42229)
I20260926 16:42:39.768239 28908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.747648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54934 (local address 127.21.12.124:42743)
0926 16:42:39.748003 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:39.748007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:39.749232 (+ 1225us) server_negotiation.cc:407] Connection header received
0926 16:42:39.749265 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:39.749267 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:39.749305 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:39.749381 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:39.754499 (+ 5118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.755292 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.758815 (+ 3523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.759125 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.759197 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.764677 (+ 5480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:39.764680 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:39.764683 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:39.764706 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:39.766445 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:39.766926 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:39.766931 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:39.766934 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:39.767031 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:39.767315 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:39.767318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:39.767319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:39.767523 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:39.767691 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:39.767959 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 16:42:39.768088 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":78,"threads_started":1}
I20260926 16:42:39.768805 27655 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.12.67@KRBTEST.COM'} at 127.21.12.67:56201
W20260926 16:42:39.769117 27656 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:42:39.771570 28773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "39e91c6cfaa7419891443717312b224d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } peers { permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 35885 } attrs { promote: false } } peers { permanent_uuid: "324ff229b53c4d59b8116721df0529be" member_type: NON_VOTER last_known_addr { host: "127.21.12.124" port: 42743 } attrs { promote: true } } }
I20260926 16:42:39.771705 28773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:39.774682 28913 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d: attempting to promote NON_VOTER 324ff229b53c4d59b8116721df0529be to VOTER
I20260926 16:42:39.774982 28913 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } peers { permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 35885 } attrs { promote: false } } peers { permanent_uuid: "324ff229b53c4d59b8116721df0529be" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 42743 } attrs { promote: false } }
I20260926 16:42:39.775943 28450 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [term 1 LEARNER]: Refusing update from remote peer 39e91c6cfaa7419891443717312b224d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 16:42:39.776309 28916 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d [LEADER]: Connected to new peer: Peer: permanent_uuid: "324ff229b53c4d59b8116721df0529be" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 42743 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 16:42:39.776979 28259 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [term 1 FOLLOWER]: Refusing update from remote peer 39e91c6cfaa7419891443717312b224d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 16:42:39.777385 28913 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 35885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 16:42:39.778155 28916 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 324ff229b53c4d59b8116721df0529be (127.21.12.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } peers { permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 35885 } attrs { promote: false } } peers { permanent_uuid: "324ff229b53c4d59b8116721df0529be" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 42743 } attrs { promote: false } } }
I20260926 16:42:39.779552 28916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "39e91c6cfaa7419891443717312b224d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } peers { permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 35885 } attrs { promote: false } } peers { permanent_uuid: "324ff229b53c4d59b8116721df0529be" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 42743 } attrs { promote: false } } }
I20260926 16:42:39.779657 28916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:39.779605 28914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "39e91c6cfaa7419891443717312b224d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } peers { permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 35885 } attrs { promote: false } } peers { permanent_uuid: "324ff229b53c4d59b8116721df0529be" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 42743 } attrs { promote: false } } }
I20260926 16:42:39.779716 28914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:39.779753 28916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "39e91c6cfaa7419891443717312b224d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } peers { permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 35885 } attrs { promote: false } } peers { permanent_uuid: "324ff229b53c4d59b8116721df0529be" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 42743 } attrs { promote: false } } }
I20260926 16:42:39.779821 28916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:39.780675 28259 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 324ff229b53c4d59b8116721df0529be (127.21.12.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } peers { permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 35885 } attrs { promote: false } } peers { permanent_uuid: "324ff229b53c4d59b8116721df0529be" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 42743 } attrs { promote: false } } }
I20260926 16:42:39.782757 28922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "39e91c6cfaa7419891443717312b224d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } peers { permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 35885 } attrs { promote: false } } peers { permanent_uuid: "324ff229b53c4d59b8116721df0529be" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 42743 } attrs { promote: false } } }
I20260926 16:42:39.782883 28922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:42:39.782830 28450 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 324ff229b53c4d59b8116721df0529be (127.21.12.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } peers { permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 35885 } attrs { promote: false } } peers { permanent_uuid: "324ff229b53c4d59b8116721df0529be" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 42743 } attrs { promote: false } } }
I20260926 16:42:39.784514 28889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "39e91c6cfaa7419891443717312b224d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } peers { permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 35885 } attrs { promote: false } } peers { permanent_uuid: "324ff229b53c4d59b8116721df0529be" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 42743 } attrs { promote: false } } }
I20260926 16:42:39.784628 28889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:42:39.793300 28828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.788184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:37069 (local address 127.0.0.1:32846)
0926 16:42:39.788302 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:39.788318 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:39.788393 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:39.789144 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:39.789148 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:39.789169 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:39.789293 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:39.789302 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.790469 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.790475 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:39.791746 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:39.791757 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.792836 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.792839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:39.792938 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.792966 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 16:42:39.793047 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 16:42:39.793160 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":36}
I20260926 16:42:39.793314 28924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.788381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32846 (local address 127.21.12.65:37069)
0926 16:42:39.788612 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:39.788628 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:39.788651 (+ 23us) server_negotiation.cc:407] Connection header received
0926 16:42:39.788730 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:39.788735 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:39.788960 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:39.789170 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:39.789397 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.790358 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.791888 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.792721 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.792829 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.793017 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:39.793085 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 16:42:39.793176 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":76,"threads_started":1}
I20260926 16:42:39.794905 28926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.788846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.67:42229 (local address 127.0.0.1:37182)
0926 16:42:39.789584 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:39.789597 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:39.789675 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:39.791055 (+ 1380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:39.791057 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:39.791073 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:39.791153 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:39.791159 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.792059 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.792063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:39.793408 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:39.793419 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.794501 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.794505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:39.794589 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.794613 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 16:42:39.794687 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 16:42:39.794756 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":3,"thread_start_us":94,"threads_started":1}
I20260926 16:42:39.795331 28927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.788846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37484 (local address 127.21.12.66:41203)
0926 16:42:39.790162 (+ 1316us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:39.790170 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:39.790189 (+ 19us) server_negotiation.cc:407] Connection header received
0926 16:42:39.790248 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:39.790252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:39.790394 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:39.790581 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:39.790824 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.791899 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.793923 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.794762 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.794857 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.794988 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:39.795104 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 16:42:39.795196 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1234,"thread_start_us":90,"threads_started":1}
I20260926 16:42:39.795655 28928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.789234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37182 (local address 127.21.12.67:42229)
0926 16:42:39.790620 (+ 1386us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:39.790633 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:39.790653 (+ 20us) server_negotiation.cc:407] Connection header received
0926 16:42:39.790722 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:39.790726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:39.790914 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:39.791076 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:39.791243 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.791937 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.793520 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.794352 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.794472 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.794613 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:39.795162 (+ 549us) server_negotiation.cc:299] Negotiation successful
0926 16:42:39.795323 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1288,"thread_start_us":102,"threads_started":1}
I20260926 16:42:39.795254 28925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.788464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:41203 (local address 127.0.0.1:37484)
0926 16:42:39.788780 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:39.788793 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:39.788868 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:39.790562 (+ 1694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:39.790565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:39.790583 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:39.790708 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:39.790715 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.792013 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.792018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:39.793786 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:39.793797 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.794872 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.794876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:39.794962 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.794983 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 16:42:39.795076 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 16:42:39.795138 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":91,"thread_start_us":94,"threads_started":1}
I20260926 16:42:39.796934 28553 tablet_service.cc:1467] Processing CreateTablet for tablet 2565b66e86f74f2ba4a56b1c096f3c62 (DEFAULT_TABLE table=test-table [id=bbb21e7110c645d68034126a799f75de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:39.798139 28553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2565b66e86f74f2ba4a56b1c096f3c62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:39.798533 28690 tablet_service.cc:1467] Processing CreateTablet for tablet 2565b66e86f74f2ba4a56b1c096f3c62 (DEFAULT_TABLE table=test-table [id=bbb21e7110c645d68034126a799f75de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:39.799221 28690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2565b66e86f74f2ba4a56b1c096f3c62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:39.804620 28929 tablet_bootstrap.cc:492] T 2565b66e86f74f2ba4a56b1c096f3c62 P 2d57671a949840679f2e9b9ec4ddb8de: Bootstrap starting.
I20260926 16:42:39.805958 28930 tablet_bootstrap.cc:492] T 2565b66e86f74f2ba4a56b1c096f3c62 P 0df678e3047b4d9ca7e774b8040f86b4: Bootstrap starting.
I20260926 16:42:39.806787 28929 tablet_bootstrap.cc:654] T 2565b66e86f74f2ba4a56b1c096f3c62 P 2d57671a949840679f2e9b9ec4ddb8de: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:39.807695 28929 log.cc:824] T 2565b66e86f74f2ba4a56b1c096f3c62 P 2d57671a949840679f2e9b9ec4ddb8de: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:39.807389 28827 tablet_service.cc:1467] Processing CreateTablet for tablet 2565b66e86f74f2ba4a56b1c096f3c62 (DEFAULT_TABLE table=test-table [id=bbb21e7110c645d68034126a799f75de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:39.808094 28930 tablet_bootstrap.cc:654] T 2565b66e86f74f2ba4a56b1c096f3c62 P 0df678e3047b4d9ca7e774b8040f86b4: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:39.808370 28827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2565b66e86f74f2ba4a56b1c096f3c62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:39.808917 28930 log.cc:824] T 2565b66e86f74f2ba4a56b1c096f3c62 P 0df678e3047b4d9ca7e774b8040f86b4: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:39.809834 28929 tablet_bootstrap.cc:492] T 2565b66e86f74f2ba4a56b1c096f3c62 P 2d57671a949840679f2e9b9ec4ddb8de: No bootstrap required, opened a new log
I20260926 16:42:39.810041 28929 ts_tablet_manager.cc:1397] T 2565b66e86f74f2ba4a56b1c096f3c62 P 2d57671a949840679f2e9b9ec4ddb8de: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 16:42:39.810838 28930 tablet_bootstrap.cc:492] T 2565b66e86f74f2ba4a56b1c096f3c62 P 0df678e3047b4d9ca7e774b8040f86b4: No bootstrap required, opened a new log
I20260926 16:42:39.811012 28930 ts_tablet_manager.cc:1397] T 2565b66e86f74f2ba4a56b1c096f3c62 P 0df678e3047b4d9ca7e774b8040f86b4: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 16:42:39.813920 28929 raft_consensus.cc:348] T 2565b66e86f74f2ba4a56b1c096f3c62 P 2d57671a949840679f2e9b9ec4ddb8de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d57671a949840679f2e9b9ec4ddb8de" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 37069 } } peers { permanent_uuid: "0df678e3047b4d9ca7e774b8040f86b4" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 41203 } } peers { permanent_uuid: "b27e9cb575dc439eae6edd24bc55de6c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42229 } }
I20260926 16:42:39.814194 28929 raft_consensus.cc:374] T 2565b66e86f74f2ba4a56b1c096f3c62 P 2d57671a949840679f2e9b9ec4ddb8de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:39.814252 28929 raft_consensus.cc:729] T 2565b66e86f74f2ba4a56b1c096f3c62 P 2d57671a949840679f2e9b9ec4ddb8de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d57671a949840679f2e9b9ec4ddb8de, State: Initialized, Role: FOLLOWER
I20260926 16:42:39.814697 28929 consensus_queue.cc:260] T 2565b66e86f74f2ba4a56b1c096f3c62 P 2d57671a949840679f2e9b9ec4ddb8de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d57671a949840679f2e9b9ec4ddb8de" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 37069 } } peers { permanent_uuid: "0df678e3047b4d9ca7e774b8040f86b4" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 41203 } } peers { permanent_uuid: "b27e9cb575dc439eae6edd24bc55de6c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42229 } }
I20260926 16:42:39.815641 28929 ts_tablet_manager.cc:1428] T 2565b66e86f74f2ba4a56b1c096f3c62 P 2d57671a949840679f2e9b9ec4ddb8de: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 16:42:39.815500 28930 raft_consensus.cc:348] T 2565b66e86f74f2ba4a56b1c096f3c62 P 0df678e3047b4d9ca7e774b8040f86b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d57671a949840679f2e9b9ec4ddb8de" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 37069 } } peers { permanent_uuid: "0df678e3047b4d9ca7e774b8040f86b4" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 41203 } } peers { permanent_uuid: "b27e9cb575dc439eae6edd24bc55de6c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42229 } }
I20260926 16:42:39.815727 28930 raft_consensus.cc:374] T 2565b66e86f74f2ba4a56b1c096f3c62 P 0df678e3047b4d9ca7e774b8040f86b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:39.815783 28930 raft_consensus.cc:729] T 2565b66e86f74f2ba4a56b1c096f3c62 P 0df678e3047b4d9ca7e774b8040f86b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0df678e3047b4d9ca7e774b8040f86b4, State: Initialized, Role: FOLLOWER
I20260926 16:42:39.815910 28620 heartbeater.cc:503] Master 127.21.12.126:34799 was elected leader, sending a full tablet report...
I20260926 16:42:39.816193 28930 consensus_queue.cc:260] T 2565b66e86f74f2ba4a56b1c096f3c62 P 0df678e3047b4d9ca7e774b8040f86b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d57671a949840679f2e9b9ec4ddb8de" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 37069 } } peers { permanent_uuid: "0df678e3047b4d9ca7e774b8040f86b4" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 41203 } } peers { permanent_uuid: "b27e9cb575dc439eae6edd24bc55de6c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42229 } }
I20260926 16:42:39.817147 28934 tablet_bootstrap.cc:492] T 2565b66e86f74f2ba4a56b1c096f3c62 P b27e9cb575dc439eae6edd24bc55de6c: Bootstrap starting.
I20260926 16:42:39.817709 28757 heartbeater.cc:503] Master 127.21.12.126:34799 was elected leader, sending a full tablet report...
I20260926 16:42:39.819218 28934 tablet_bootstrap.cc:654] T 2565b66e86f74f2ba4a56b1c096f3c62 P b27e9cb575dc439eae6edd24bc55de6c: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:39.819974 28934 log.cc:824] T 2565b66e86f74f2ba4a56b1c096f3c62 P b27e9cb575dc439eae6edd24bc55de6c: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:39.821213 28930 ts_tablet_manager.cc:1428] T 2565b66e86f74f2ba4a56b1c096f3c62 P 0df678e3047b4d9ca7e774b8040f86b4: Time spent starting tablet: real 0.010s user 0.004s sys 0.003s
I20260926 16:42:39.823925 28934 tablet_bootstrap.cc:492] T 2565b66e86f74f2ba4a56b1c096f3c62 P b27e9cb575dc439eae6edd24bc55de6c: No bootstrap required, opened a new log
I20260926 16:42:39.824177 28934 ts_tablet_manager.cc:1397] T 2565b66e86f74f2ba4a56b1c096f3c62 P b27e9cb575dc439eae6edd24bc55de6c: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
W20260926 16:42:39.825403 28622 tablet.cc:2369] T 2565b66e86f74f2ba4a56b1c096f3c62 P 2d57671a949840679f2e9b9ec4ddb8de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:42:39.827538 28934 raft_consensus.cc:348] T 2565b66e86f74f2ba4a56b1c096f3c62 P b27e9cb575dc439eae6edd24bc55de6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d57671a949840679f2e9b9ec4ddb8de" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 37069 } } peers { permanent_uuid: "0df678e3047b4d9ca7e774b8040f86b4" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 41203 } } peers { permanent_uuid: "b27e9cb575dc439eae6edd24bc55de6c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42229 } }
I20260926 16:42:39.827767 28934 raft_consensus.cc:374] T 2565b66e86f74f2ba4a56b1c096f3c62 P b27e9cb575dc439eae6edd24bc55de6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:39.827826 28934 raft_consensus.cc:729] T 2565b66e86f74f2ba4a56b1c096f3c62 P b27e9cb575dc439eae6edd24bc55de6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b27e9cb575dc439eae6edd24bc55de6c, State: Initialized, Role: FOLLOWER
I20260926 16:42:39.828200 28934 consensus_queue.cc:260] T 2565b66e86f74f2ba4a56b1c096f3c62 P b27e9cb575dc439eae6edd24bc55de6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d57671a949840679f2e9b9ec4ddb8de" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 37069 } } peers { permanent_uuid: "0df678e3047b4d9ca7e774b8040f86b4" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 41203 } } peers { permanent_uuid: "b27e9cb575dc439eae6edd24bc55de6c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42229 } }
I20260926 16:42:39.828823 28896 heartbeater.cc:503] Master 127.21.12.126:34799 was elected leader, sending a full tablet report...
I20260926 16:42:39.829130 28934 ts_tablet_manager.cc:1428] T 2565b66e86f74f2ba4a56b1c096f3c62 P b27e9cb575dc439eae6edd24bc55de6c: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260926 16:42:39.890550 28935 raft_consensus.cc:482] T 2565b66e86f74f2ba4a56b1c096f3c62 P 0df678e3047b4d9ca7e774b8040f86b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:42:39.890770 28935 raft_consensus.cc:504] T 2565b66e86f74f2ba4a56b1c096f3c62 P 0df678e3047b4d9ca7e774b8040f86b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d57671a949840679f2e9b9ec4ddb8de" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 37069 } } peers { permanent_uuid: "0df678e3047b4d9ca7e774b8040f86b4" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 41203 } } peers { permanent_uuid: "b27e9cb575dc439eae6edd24bc55de6c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42229 } }
I20260926 16:42:39.891991 28935 leader_election.cc:288] T 2565b66e86f74f2ba4a56b1c096f3c62 P 0df678e3047b4d9ca7e774b8040f86b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d57671a949840679f2e9b9ec4ddb8de (127.21.12.65:37069), b27e9cb575dc439eae6edd24bc55de6c (127.21.12.67:42229)
I20260926 16:42:39.896040 28928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.892522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:39045 (local address 127.21.12.67:42229)
0926 16:42:39.892692 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:39.892698 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:39.892712 (+ 14us) server_negotiation.cc:407] Connection header received
0926 16:42:39.892772 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:39.892774 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:39.892810 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:39.892912 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:39.893194 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.893813 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.895057 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.895494 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.895647 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.895702 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:39.895851 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 16:42:39.895925 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 16:42:39.896063 28760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.892233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.67:42229 (local address 127.21.12.66:39045)
0926 16:42:39.892386 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:39.892401 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:39.892485 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:39.892924 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:39.892927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:39.892942 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:39.893055 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:39.893061 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.893926 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.893933 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:39.894934 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:39.894943 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.895615 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.895618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:39.895712 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.895741 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 16:42:39.895838 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 16:42:39.895910 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"mutex_wait_us":19}
I20260926 16:42:39.896624 28762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.892164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:37069 (local address 127.21.12.66:49505)
0926 16:42:39.892279 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:39.892296 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:39.892386 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:39.892699 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:39.892702 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:39.892718 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:39.892827 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:39.892832 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.893626 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.893632 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:39.894861 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:39.894874 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.896202 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.896206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:39.896291 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.896333 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 16:42:39.896418 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 16:42:39.896488 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":23}
I20260926 16:42:39.896668 28924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.892237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:49505 (local address 127.21.12.65:37069)
0926 16:42:39.892381 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:39.892386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:39.892402 (+ 16us) server_negotiation.cc:407] Connection header received
0926 16:42:39.892513 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:39.892520 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:39.892575 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:39.892657 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:39.892939 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.893500 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.894969 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.895492 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.895613 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.895677 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:39.896450 (+ 773us) server_negotiation.cc:299] Negotiation successful
0926 16:42:39.896527 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 16:42:39.896771 28848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2565b66e86f74f2ba4a56b1c096f3c62" candidate_uuid: "0df678e3047b4d9ca7e774b8040f86b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b27e9cb575dc439eae6edd24bc55de6c" is_pre_election: true
I20260926 16:42:39.897082 28848 raft_consensus.cc:2454] T 2565b66e86f74f2ba4a56b1c096f3c62 P b27e9cb575dc439eae6edd24bc55de6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0df678e3047b4d9ca7e774b8040f86b4 in term 0.
I20260926 16:42:39.897181 28573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2565b66e86f74f2ba4a56b1c096f3c62" candidate_uuid: "0df678e3047b4d9ca7e774b8040f86b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d57671a949840679f2e9b9ec4ddb8de" is_pre_election: true
I20260926 16:42:39.897455 28573 raft_consensus.cc:2454] T 2565b66e86f74f2ba4a56b1c096f3c62 P 2d57671a949840679f2e9b9ec4ddb8de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0df678e3047b4d9ca7e774b8040f86b4 in term 0.
I20260926 16:42:39.897565 28645 leader_election.cc:302] T 2565b66e86f74f2ba4a56b1c096f3c62 P 0df678e3047b4d9ca7e774b8040f86b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0df678e3047b4d9ca7e774b8040f86b4, b27e9cb575dc439eae6edd24bc55de6c; no voters:
I20260926 16:42:39.897850 28935 raft_consensus.cc:2781] T 2565b66e86f74f2ba4a56b1c096f3c62 P 0df678e3047b4d9ca7e774b8040f86b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:42:39.897929 28935 raft_consensus.cc:482] T 2565b66e86f74f2ba4a56b1c096f3c62 P 0df678e3047b4d9ca7e774b8040f86b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:42:39.897981 28935 raft_consensus.cc:3037] T 2565b66e86f74f2ba4a56b1c096f3c62 P 0df678e3047b4d9ca7e774b8040f86b4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:39.899230 28935 raft_consensus.cc:504] T 2565b66e86f74f2ba4a56b1c096f3c62 P 0df678e3047b4d9ca7e774b8040f86b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d57671a949840679f2e9b9ec4ddb8de" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 37069 } } peers { permanent_uuid: "0df678e3047b4d9ca7e774b8040f86b4" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 41203 } } peers { permanent_uuid: "b27e9cb575dc439eae6edd24bc55de6c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42229 } }
I20260926 16:42:39.899798 28935 leader_election.cc:288] T 2565b66e86f74f2ba4a56b1c096f3c62 P 0df678e3047b4d9ca7e774b8040f86b4 [CANDIDATE]: Term 1 election: Requested vote from peers 2d57671a949840679f2e9b9ec4ddb8de (127.21.12.65:37069), b27e9cb575dc439eae6edd24bc55de6c (127.21.12.67:42229)
I20260926 16:42:39.899890 28848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2565b66e86f74f2ba4a56b1c096f3c62" candidate_uuid: "0df678e3047b4d9ca7e774b8040f86b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b27e9cb575dc439eae6edd24bc55de6c"
I20260926 16:42:39.899889 28573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2565b66e86f74f2ba4a56b1c096f3c62" candidate_uuid: "0df678e3047b4d9ca7e774b8040f86b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d57671a949840679f2e9b9ec4ddb8de"
I20260926 16:42:39.900020 28848 raft_consensus.cc:3037] T 2565b66e86f74f2ba4a56b1c096f3c62 P b27e9cb575dc439eae6edd24bc55de6c [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:39.900033 28573 raft_consensus.cc:3037] T 2565b66e86f74f2ba4a56b1c096f3c62 P 2d57671a949840679f2e9b9ec4ddb8de [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:39.901327 28848 raft_consensus.cc:2454] T 2565b66e86f74f2ba4a56b1c096f3c62 P b27e9cb575dc439eae6edd24bc55de6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0df678e3047b4d9ca7e774b8040f86b4 in term 1.
I20260926 16:42:39.901327 28573 raft_consensus.cc:2454] T 2565b66e86f74f2ba4a56b1c096f3c62 P 2d57671a949840679f2e9b9ec4ddb8de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0df678e3047b4d9ca7e774b8040f86b4 in term 1.
I20260926 16:42:39.901674 28646 leader_election.cc:302] T 2565b66e86f74f2ba4a56b1c096f3c62 P 0df678e3047b4d9ca7e774b8040f86b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0df678e3047b4d9ca7e774b8040f86b4, 2d57671a949840679f2e9b9ec4ddb8de; no voters:
I20260926 16:42:39.901948 28935 raft_consensus.cc:2781] T 2565b66e86f74f2ba4a56b1c096f3c62 P 0df678e3047b4d9ca7e774b8040f86b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:39.902176 28935 raft_consensus.cc:686] T 2565b66e86f74f2ba4a56b1c096f3c62 P 0df678e3047b4d9ca7e774b8040f86b4 [term 1 LEADER]: Becoming Leader. State: Replica: 0df678e3047b4d9ca7e774b8040f86b4, State: Running, Role: LEADER
I20260926 16:42:39.902607 28935 consensus_queue.cc:237] T 2565b66e86f74f2ba4a56b1c096f3c62 P 0df678e3047b4d9ca7e774b8040f86b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d57671a949840679f2e9b9ec4ddb8de" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 37069 } } peers { permanent_uuid: "0df678e3047b4d9ca7e774b8040f86b4" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 41203 } } peers { permanent_uuid: "b27e9cb575dc439eae6edd24bc55de6c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42229 } }
I20260926 16:42:39.906804 27655 catalog_manager.cc:5495] T 2565b66e86f74f2ba4a56b1c096f3c62 P 0df678e3047b4d9ca7e774b8040f86b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0df678e3047b4d9ca7e774b8040f86b4 (127.21.12.66). New cstate: current_term: 1 leader_uuid: "0df678e3047b4d9ca7e774b8040f86b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d57671a949840679f2e9b9ec4ddb8de" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 37069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0df678e3047b4d9ca7e774b8040f86b4" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 41203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b27e9cb575dc439eae6edd24bc55de6c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42229 } health_report { overall_health: UNKNOWN } } }
I20260926 16:42:39.943341 28927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.939220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37496 (local address 127.21.12.66:41203)
0926 16:42:39.939374 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:39.939379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:39.939396 (+ 17us) server_negotiation.cc:407] Connection header received
0926 16:42:39.939649 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:39.939652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:39.939711 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:39.939833 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:42:39.940095 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.940649 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.941596 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.941923 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.942046 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.942397 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:42:39.942910 (+ 513us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:42:39.943047 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:39.943125 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 16:42:39.943184 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 16:42:39.949064 28848 raft_consensus.cc:1261] T 2565b66e86f74f2ba4a56b1c096f3c62 P b27e9cb575dc439eae6edd24bc55de6c [term 1 FOLLOWER]: Refusing update from remote peer 0df678e3047b4d9ca7e774b8040f86b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:42:39.949121 28573 raft_consensus.cc:1261] T 2565b66e86f74f2ba4a56b1c096f3c62 P 2d57671a949840679f2e9b9ec4ddb8de [term 1 FOLLOWER]: Refusing update from remote peer 0df678e3047b4d9ca7e774b8040f86b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:42:39.949894 28938 consensus_queue.cc:1035] T 2565b66e86f74f2ba4a56b1c096f3c62 P 0df678e3047b4d9ca7e774b8040f86b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b27e9cb575dc439eae6edd24bc55de6c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:42:39.950130 28935 consensus_queue.cc:1035] T 2565b66e86f74f2ba4a56b1c096f3c62 P 0df678e3047b4d9ca7e774b8040f86b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d57671a949840679f2e9b9ec4ddb8de" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 37069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:42:39.964535 28942 mvcc.cc:205] Tried to move back new op lower bound from 7333646171944255488 to 7333646171763798016. Current Snapshot: MvccSnapshot[applied={T|T < 7333646171944255488}]
I20260926 16:42:39.967087 27655 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:42:39.967248 27655 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:42:39.971505 27655 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 39e91c6cfaa7419891443717312b224d: Sending DeleteTablet for 3 replicas of tablet 2565b66e86f74f2ba4a56b1c096f3c62
I20260926 16:42:39.972175 28553 tablet_service.cc:1514] Processing DeleteTablet for tablet 2565b66e86f74f2ba4a56b1c096f3c62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:39 UTC) from {username='oryx', principal='oryx/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:32846
I20260926 16:42:39.973331 28690 tablet_service.cc:1514] Processing DeleteTablet for tablet 2565b66e86f74f2ba4a56b1c096f3c62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:39 UTC) from {username='oryx', principal='oryx/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:37484
I20260926 16:42:39.973343 28827 tablet_service.cc:1514] Processing DeleteTablet for tablet 2565b66e86f74f2ba4a56b1c096f3c62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:39 UTC) from {username='oryx', principal='oryx/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:37182
I20260926 16:42:39.973840 28952 tablet_replica.cc:330] T 2565b66e86f74f2ba4a56b1c096f3c62 P b27e9cb575dc439eae6edd24bc55de6c: stopping tablet replica
I20260926 16:42:39.973927 28953 tablet_replica.cc:330] T 2565b66e86f74f2ba4a56b1c096f3c62 P 0df678e3047b4d9ca7e774b8040f86b4: stopping tablet replica
I20260926 16:42:39.974205 28953 raft_consensus.cc:2229] T 2565b66e86f74f2ba4a56b1c096f3c62 P 0df678e3047b4d9ca7e774b8040f86b4 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:42:39.974552 28953 raft_consensus.cc:2258] T 2565b66e86f74f2ba4a56b1c096f3c62 P 0df678e3047b4d9ca7e774b8040f86b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:42:39.975066 28952 raft_consensus.cc:2229] T 2565b66e86f74f2ba4a56b1c096f3c62 P b27e9cb575dc439eae6edd24bc55de6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:42:39.975365 28952 raft_consensus.cc:2258] T 2565b66e86f74f2ba4a56b1c096f3c62 P b27e9cb575dc439eae6edd24bc55de6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:42:39.976334 28953 ts_tablet_manager.cc:1905] T 2565b66e86f74f2ba4a56b1c096f3c62 P 0df678e3047b4d9ca7e774b8040f86b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:42:39.976562 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 27624
I20260926 16:42:39.980048 28953 ts_tablet_manager.cc:1918] T 2565b66e86f74f2ba4a56b1c096f3c62 P 0df678e3047b4d9ca7e774b8040f86b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:42:39.980149 28953 log.cc:1196] T 2565b66e86f74f2ba4a56b1c096f3c62 P 0df678e3047b4d9ca7e774b8040f86b4: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-1/wal/wals/2565b66e86f74f2ba4a56b1c096f3c62
I20260926 16:42:39.980520 28953 ts_tablet_manager.cc:1939] T 2565b66e86f74f2ba4a56b1c096f3c62 P 0df678e3047b4d9ca7e774b8040f86b4: Deleting consensus metadata
I20260926 16:42:39.983702 28951 tablet_replica.cc:330] T 2565b66e86f74f2ba4a56b1c096f3c62 P 2d57671a949840679f2e9b9ec4ddb8de: stopping tablet replica
I20260926 16:42:39.984040 28951 raft_consensus.cc:2229] T 2565b66e86f74f2ba4a56b1c096f3c62 P 2d57671a949840679f2e9b9ec4ddb8de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:42:39.984375 28951 raft_consensus.cc:2258] T 2565b66e86f74f2ba4a56b1c096f3c62 P 2d57671a949840679f2e9b9ec4ddb8de [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 16:42:39.985986 28645 connection.cc:535] server connection from 127.0.0.1:37484 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 16:42:39.986095 28951 ts_tablet_manager.cc:1905] T 2565b66e86f74f2ba4a56b1c096f3c62 P 2d57671a949840679f2e9b9ec4ddb8de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:42:39.987150 28952 ts_tablet_manager.cc:1905] T 2565b66e86f74f2ba4a56b1c096f3c62 P b27e9cb575dc439eae6edd24bc55de6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:42:39.988089 28910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.974936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40076 (local address 127.21.12.125:35885)
0926 16:42:39.975327 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:39.975331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:39.977520 (+ 2189us) server_negotiation.cc:407] Connection header received
0926 16:42:39.977712 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:39.977715 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:39.977786 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:39.977839 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:39.978940 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.979811 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.981283 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.981637 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.981714 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.983878 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:39.983882 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:39.983884 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:39.983909 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:39.985924 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:39.986545 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:39.986549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:39.986550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:39.986639 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:39.987063 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:39.987066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:39.987068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:39.987281 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:39.987394 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:39.987768 (+ 374us) server_negotiation.cc:299] Negotiation successful
0926 16:42:39.987894 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309}
I20260926 16:42:39.988950 28951 ts_tablet_manager.cc:1918] T 2565b66e86f74f2ba4a56b1c096f3c62 P 2d57671a949840679f2e9b9ec4ddb8de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:42:39.990262 28952 ts_tablet_manager.cc:1918] T 2565b66e86f74f2ba4a56b1c096f3c62 P b27e9cb575dc439eae6edd24bc55de6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:42:39.990389 28952 log.cc:1196] T 2565b66e86f74f2ba4a56b1c096f3c62 P b27e9cb575dc439eae6edd24bc55de6c: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-2/wal/wals/2565b66e86f74f2ba4a56b1c096f3c62
I20260926 16:42:39.990764 28952 ts_tablet_manager.cc:1939] T 2565b66e86f74f2ba4a56b1c096f3c62 P b27e9cb575dc439eae6edd24bc55de6c: Deleting consensus metadata
W20260926 16:42:39.991945 28781 connection.cc:441] server connection from 127.0.0.1:37182 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37182 (request call id 1) could send its response
I20260926 16:42:39.989043 28951 log.cc:1196] T 2565b66e86f74f2ba4a56b1c096f3c62 P 2d57671a949840679f2e9b9ec4ddb8de: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-0/wal/wals/2565b66e86f74f2ba4a56b1c096f3c62
I20260926 16:42:39.994939 28951 ts_tablet_manager.cc:1939] T 2565b66e86f74f2ba4a56b1c096f3c62 P 2d57671a949840679f2e9b9ec4ddb8de: Deleting consensus metadata
W20260926 16:42:39.996120 28508 connection.cc:441] server connection from 127.0.0.1:32846 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32846 (request call id 1) could send its response
I20260926 16:42:40.000941 28908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.975460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54944 (local address 127.21.12.124:42743)
0926 16:42:39.975557 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:39.975562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:39.989504 (+ 13942us) server_negotiation.cc:407] Connection header received
0926 16:42:39.989674 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:39.989678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:39.989728 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:39.989807 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:39.990696 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.991505 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.992666 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.993021 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.993099 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.994933 (+ 1834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:39.994937 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:39.994940 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:39.994966 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:39.997372 (+ 2406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:39.999370 (+ 1998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:39.999374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:39.999376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:39.999452 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:39.999735 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:39.999738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:39.999740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:39.999936 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:40.000059 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:40.000665 (+ 606us) server_negotiation.cc:299] Negotiation successful
0926 16:42:40.000782 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":38}
I20260926 16:42:40.000967 28910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.990117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40082 (local address 127.21.12.125:35885)
0926 16:42:39.990230 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:39.990236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:39.992339 (+ 2103us) server_negotiation.cc:407] Connection header received
0926 16:42:39.992522 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:39.992525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:39.992569 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:39.992618 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:39.993579 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.994366 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.995604 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.995964 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.996046 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.997676 (+ 1630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:39.997679 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:39.997682 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:39.997704 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:39.999158 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:39.999746 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:39.999750 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:39.999752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:39.999834 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:40.000113 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:40.000117 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:40.000119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:40.000280 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:40.000408 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:40.000679 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 16:42:40.000783 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 16:42:40.002306 28623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:39.989408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54946 (local address 127.21.12.124:42743)
0926 16:42:39.989520 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:39.989524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:39.991144 (+ 1620us) server_negotiation.cc:407] Connection header received
0926 16:42:39.991302 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:39.991305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:39.991347 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:39.991392 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:39.992691 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.993480 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.994771 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:39.995110 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:39.995179 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:39.996913 (+ 1734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:39.996918 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:39.996920 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:39.996957 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:39.998849 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:39.999746 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:39.999750 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:39.999752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:40.000128 (+ 376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:40.000593 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:40.000598 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:40.000600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:40.000799 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:40.000915 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:40.002057 (+ 1142us) server_negotiation.cc:299] Negotiation successful
0926 16:42:40.002161 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 16:42:40.962066 28962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:40.961522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34799 (local address 127.21.12.67:37917)
0926 16:42:40.961828 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:40.961942 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:34799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":71,"threads_started":1}
W20260926 16:42:40.962462 28896 heartbeater.cc:644] Failed to heartbeat to 127.21.12.126:34799 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.12.126:34799: connect: Connection refused (error 111)
I20260926 16:42:40.964779 28963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:40.964372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34799 (local address 127.21.12.66:60801)
0926 16:42:40.964587 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:40.964677 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:34799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":89,"threads_started":1}
W20260926 16:42:40.965222 28757 heartbeater.cc:644] Failed to heartbeat to 127.21.12.126:34799 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.12.126:34799: connect: Connection refused (error 111)
I20260926 16:42:40.965881 28964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:40.965371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34799 (local address 127.21.12.65:55901)
0926 16:42:40.965681 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:40.965791 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:34799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":66,"threads_started":1}
W20260926 16:42:40.966126 28620 heartbeater.cc:644] Failed to heartbeat to 127.21.12.126:34799 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.12.126:34799: connect: Connection refused (error 111)
I20260926 16:42:41.472394 28965 raft_consensus.cc:482] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 39e91c6cfaa7419891443717312b224d)
I20260926 16:42:41.472584 28965 raft_consensus.cc:504] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } peers { permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 35885 } attrs { promote: false } } peers { permanent_uuid: "324ff229b53c4d59b8116721df0529be" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 42743 } attrs { promote: false } }
I20260926 16:42:41.473016 28966 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 39e91c6cfaa7419891443717312b224d)
I20260926 16:42:41.473203 28966 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } peers { permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 35885 } attrs { promote: false } } peers { permanent_uuid: "324ff229b53c4d59b8116721df0529be" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 42743 } attrs { promote: false } }
I20260926 16:42:41.473721 28965 leader_election.cc:288] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 39e91c6cfaa7419891443717312b224d (127.21.12.126:34799), 1f70fdb997ce4656b970b08582ad2556 (127.21.12.125:35885)
I20260926 16:42:41.474156 28966 leader_election.cc:288] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 39e91c6cfaa7419891443717312b224d (127.21.12.126:34799), 324ff229b53c4d59b8116721df0529be (127.21.12.124:42743)
I20260926 16:42:41.474741 28970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:41.474209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34799 (local address 127.0.0.1:54788)
0926 16:42:41.474572 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:41.474640 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:34799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":68,"threads_started":1}
W20260926 16:42:41.474987 28241 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.12.126:34799: connect: Connection refused (error 111)
I20260926 16:42:41.474851 28967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:41.473883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34799 (local address 127.0.0.1:54780)
0926 16:42:41.474469 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:41.474765 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:34799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":53,"threads_started":1}
W20260926 16:42:41.475471 28424 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.12.126:34799: connect: Connection refused (error 111)
I20260926 16:42:41.475965 28970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:41.475661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34799 (local address 127.0.0.1:54798)
0926 16:42:41.475839 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:41.475891 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:34799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20260926 16:42:41.476151 28241 leader_election.cc:334] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 39e91c6cfaa7419891443717312b224d (127.21.12.126:34799): Network error: Client connection negotiation failed: client connection to 127.21.12.126:34799: connect: Connection refused (error 111)
I20260926 16:42:41.476321 28967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:41.476082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34799 (local address 127.0.0.1:54814)
0926 16:42:41.476214 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:41.476257 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:34799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260926 16:42:41.476459 28424 leader_election.cc:334] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 39e91c6cfaa7419891443717312b224d (127.21.12.126:34799): Network error: Client connection negotiation failed: client connection to 127.21.12.126:34799: connect: Connection refused (error 111)
I20260926 16:42:41.479010 28971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:41.474358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:42743 (local address 127.0.0.1:54952)
0926 16:42:41.474777 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:41.474790 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:41.474874 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:41.475215 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:41.475218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:41.475232 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:41.475328 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:41.475338 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:41.476313 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:41.476316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:41.477598 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:41.477606 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:41.478541 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:41.478545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:41.478665 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:41.478691 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 16:42:41.478771 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 16:42:41.478845 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":1,"thread_start_us":53,"threads_started":1}
I20260926 16:42:41.479012 28968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:41.473960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.125:35885 (local address 127.0.0.1:40086)
0926 16:42:41.474603 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:41.474618 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:41.474764 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:41.475132 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:41.475134 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:41.475157 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:41.475240 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:41.475246 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:41.476316 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:41.476319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:41.477596 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:41.477607 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:41.478545 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:41.478548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:41.478665 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:41.478691 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 16:42:41.478771 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 16:42:41.478845 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"mutex_wait_us":1,"spinlock_wait_cycles":896,"thread_start_us":64,"threads_started":1}
I20260926 16:42:41.479017 28972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:41.474469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54952 (local address 127.21.12.124:42743)
0926 16:42:41.474666 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:41.474672 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:41.474950 (+ 278us) server_negotiation.cc:407] Connection header received
0926 16:42:41.474995 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:41.474997 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:41.475044 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:41.475116 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:41.475436 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:41.476196 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:41.477749 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:41.478429 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:41.478541 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:41.478655 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:41.478799 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 16:42:41.478875 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":132,"thread_start_us":69,"threads_started":1}
I20260926 16:42:41.479040 28969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:41.474062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40086 (local address 127.21.12.125:35885)
0926 16:42:41.474455 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:41.474460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:41.474684 (+ 224us) server_negotiation.cc:407] Connection header received
0926 16:42:41.474886 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:41.474890 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:41.474938 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:41.474990 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:41.475351 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:41.476184 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:41.477749 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:41.478416 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:41.478541 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:41.478645 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:41.478799 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 16:42:41.478875 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":319,"thread_start_us":56,"threads_started":1}
I20260926 16:42:41.479604 28259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "324ff229b53c4d59b8116721df0529be" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1f70fdb997ce4656b970b08582ad2556" is_pre_election: true
I20260926 16:42:41.479604 28450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1f70fdb997ce4656b970b08582ad2556" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "324ff229b53c4d59b8116721df0529be" is_pre_election: true
I20260926 16:42:41.479805 28450 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f70fdb997ce4656b970b08582ad2556 in term 1.
I20260926 16:42:41.479805 28259 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 324ff229b53c4d59b8116721df0529be in term 1.
I20260926 16:42:41.480165 28425 leader_election.cc:302] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f70fdb997ce4656b970b08582ad2556, 324ff229b53c4d59b8116721df0529be; no voters: 39e91c6cfaa7419891443717312b224d
I20260926 16:42:41.480209 28243 leader_election.cc:302] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f70fdb997ce4656b970b08582ad2556, 324ff229b53c4d59b8116721df0529be; no voters: 39e91c6cfaa7419891443717312b224d
I20260926 16:42:41.480386 28965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 16:42:41.480459 28965 raft_consensus.cc:482] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [term 1 FOLLOWER]: Starting leader election (detected failure of leader 39e91c6cfaa7419891443717312b224d)
I20260926 16:42:41.480516 28965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [term 1 FOLLOWER]: Advancing to term 2
I20260926 16:42:41.481525 28965 raft_consensus.cc:504] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } peers { permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 35885 } attrs { promote: false } } peers { permanent_uuid: "324ff229b53c4d59b8116721df0529be" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 42743 } attrs { promote: false } }
I20260926 16:42:41.481842 28966 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 16:42:41.481909 28966 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 39e91c6cfaa7419891443717312b224d)
I20260926 16:42:41.481957 28966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [term 1 FOLLOWER]: Advancing to term 2
I20260926 16:42:41.482008 28965 leader_election.cc:288] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [CANDIDATE]: Term 2 election: Requested vote from peers 39e91c6cfaa7419891443717312b224d (127.21.12.126:34799), 1f70fdb997ce4656b970b08582ad2556 (127.21.12.125:35885)
I20260926 16:42:41.482311 28968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:41.482112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34799 (local address 127.0.0.1:54816)
0926 16:42:41.482204 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:41.482256 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:34799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 16:42:41.482280 28259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "324ff229b53c4d59b8116721df0529be" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1f70fdb997ce4656b970b08582ad2556"
I20260926 16:42:41.483049 28968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:41.482822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34799 (local address 127.0.0.1:54822)
0926 16:42:41.482921 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:41.482971 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:34799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 16:42:41.483166 28424 leader_election.cc:334] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 39e91c6cfaa7419891443717312b224d (127.21.12.126:34799): Network error: Client connection negotiation failed: client connection to 127.21.12.126:34799: connect: Connection refused (error 111)
I20260926 16:42:41.483201 28966 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } peers { permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 35885 } attrs { promote: false } } peers { permanent_uuid: "324ff229b53c4d59b8116721df0529be" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 42743 } attrs { promote: false } }
I20260926 16:42:41.483448 28259 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 324ff229b53c4d59b8116721df0529be in current term 2: Already voted for candidate 1f70fdb997ce4656b970b08582ad2556 in this term.
I20260926 16:42:41.483624 28966 leader_election.cc:288] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [CANDIDATE]: Term 2 election: Requested vote from peers 39e91c6cfaa7419891443717312b224d (127.21.12.126:34799), 324ff229b53c4d59b8116721df0529be (127.21.12.124:42743)
I20260926 16:42:41.483945 28450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1f70fdb997ce4656b970b08582ad2556" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "324ff229b53c4d59b8116721df0529be"
I20260926 16:42:41.484118 28450 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1f70fdb997ce4656b970b08582ad2556 in current term 2: Already voted for candidate 324ff229b53c4d59b8116721df0529be in this term.
I20260926 16:42:41.484201 28971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:41.483886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34799 (local address 127.0.0.1:54836)
0926 16:42:41.483984 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:41.484099 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:34799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 16:42:41.484166 28425 leader_election.cc:302] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 324ff229b53c4d59b8116721df0529be; no voters: 1f70fdb997ce4656b970b08582ad2556, 39e91c6cfaa7419891443717312b224d
I20260926 16:42:41.484445 28965 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260926 16:42:41.484794 28971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:41.484628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34799 (local address 127.0.0.1:54852)
0926 16:42:41.484707 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:41.484749 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:34799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260926 16:42:41.484880 28241 leader_election.cc:334] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 39e91c6cfaa7419891443717312b224d (127.21.12.126:34799): Network error: Client connection negotiation failed: client connection to 127.21.12.126:34799: connect: Connection refused (error 111)
I20260926 16:42:41.484969 28241 leader_election.cc:302] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f70fdb997ce4656b970b08582ad2556; no voters: 324ff229b53c4d59b8116721df0529be, 39e91c6cfaa7419891443717312b224d
I20260926 16:42:41.485093 28966 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260926 16:42:41.963739 28975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:41.963217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34799 (local address 127.21.12.67:55935)
0926 16:42:41.963510 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:41.963625 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:34799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":84,"threads_started":1}
I20260926 16:42:41.966383 28976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:41.965814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34799 (local address 127.21.12.66:45299)
0926 16:42:41.966200 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:41.966266 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:34799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":114,"threads_started":1}
I20260926 16:42:41.966989 28977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:41.966637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34799 (local address 127.21.12.65:46407)
0926 16:42:41.966844 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:41.966901 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:34799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":77,"threads_started":1}
I20260926 16:42:42.965292 28978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:42.964811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34799 (local address 127.21.12.67:44681)
0926 16:42:42.965109 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:42.965181 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:34799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":77,"threads_started":1}
I20260926 16:42:42.968041 28979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:42.967615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34799 (local address 127.21.12.66:35705)
0926 16:42:42.967841 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:42.967911 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:34799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":67,"threads_started":1}
I20260926 16:42:42.968634 28980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:42.967735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34799 (local address 127.21.12.65:37851)
0926 16:42:42.968471 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:42.968534 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:34799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":67,"threads_started":1}
I20260926 16:42:43.374300 28981 raft_consensus.cc:482] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:42:43.374436 28981 raft_consensus.cc:504] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } peers { permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 35885 } attrs { promote: false } } peers { permanent_uuid: "324ff229b53c4d59b8116721df0529be" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 42743 } attrs { promote: false } }
I20260926 16:42:43.374925 28981 leader_election.cc:288] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39e91c6cfaa7419891443717312b224d (127.21.12.126:34799), 1f70fdb997ce4656b970b08582ad2556 (127.21.12.125:35885)
I20260926 16:42:43.375344 28259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "324ff229b53c4d59b8116721df0529be" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1f70fdb997ce4656b970b08582ad2556" is_pre_election: true
I20260926 16:42:43.375511 28259 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 324ff229b53c4d59b8116721df0529be in term 2.
I20260926 16:42:43.375537 28982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:43.375113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34799 (local address 127.0.0.1:55078)
0926 16:42:43.375348 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:43.375411 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:34799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":83,"threads_started":1}
I20260926 16:42:43.375882 28425 leader_election.cc:302] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f70fdb997ce4656b970b08582ad2556, 324ff229b53c4d59b8116721df0529be; no voters:
I20260926 16:42:43.376089 28981 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260926 16:42:43.376149 28981 raft_consensus.cc:482] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:42:43.376179 28981 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [term 2 FOLLOWER]: Advancing to term 3
I20260926 16:42:43.376660 28982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:43.376374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34799 (local address 127.0.0.1:55090)
0926 16:42:43.376524 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:43.376595 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:34799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260926 16:42:43.376804 28424 leader_election.cc:334] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 39e91c6cfaa7419891443717312b224d (127.21.12.126:34799): Network error: Client connection negotiation failed: client connection to 127.21.12.126:34799: connect: Connection refused (error 111)
I20260926 16:42:43.377514 28981 raft_consensus.cc:504] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } peers { permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 35885 } attrs { promote: false } } peers { permanent_uuid: "324ff229b53c4d59b8116721df0529be" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 42743 } attrs { promote: false } }
I20260926 16:42:43.377918 28981 leader_election.cc:288] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [CANDIDATE]: Term 3 election: Requested vote from peers 39e91c6cfaa7419891443717312b224d (127.21.12.126:34799), 1f70fdb997ce4656b970b08582ad2556 (127.21.12.125:35885)
I20260926 16:42:43.378228 28982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:43.378023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34799 (local address 127.0.0.1:55092)
0926 16:42:43.378118 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:43.378165 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:34799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 16:42:43.378268 28259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "324ff229b53c4d59b8116721df0529be" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1f70fdb997ce4656b970b08582ad2556"
I20260926 16:42:43.378374 28259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [term 2 FOLLOWER]: Advancing to term 3
I20260926 16:42:43.378906 28982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:43.378700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34799 (local address 127.0.0.1:55108)
0926 16:42:43.378807 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:43.378855 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:34799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 16:42:43.379053 28424 leader_election.cc:334] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 39e91c6cfaa7419891443717312b224d (127.21.12.126:34799): Network error: Client connection negotiation failed: client connection to 127.21.12.126:34799: connect: Connection refused (error 111)
I20260926 16:42:43.379601 28259 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 324ff229b53c4d59b8116721df0529be in term 3.
I20260926 16:42:43.379873 28425 leader_election.cc:302] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f70fdb997ce4656b970b08582ad2556, 324ff229b53c4d59b8116721df0529be; no voters: 39e91c6cfaa7419891443717312b224d
I20260926 16:42:43.380031 28981 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [term 3 FOLLOWER]: Leader election won for term 3
I20260926 16:42:43.380235 28981 raft_consensus.cc:686] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [term 3 LEADER]: Becoming Leader. State: Replica: 324ff229b53c4d59b8116721df0529be, State: Running, Role: LEADER
I20260926 16:42:43.380512 28981 consensus_queue.cc:237] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } peers { permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 35885 } attrs { promote: false } } peers { permanent_uuid: "324ff229b53c4d59b8116721df0529be" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 42743 } attrs { promote: false } }
I20260926 16:42:43.381866 28984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 324ff229b53c4d59b8116721df0529be. Latest consensus state: current_term: 3 leader_uuid: "324ff229b53c4d59b8116721df0529be" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } peers { permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 35885 } attrs { promote: false } } peers { permanent_uuid: "324ff229b53c4d59b8116721df0529be" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 42743 } attrs { promote: false } } }
I20260926 16:42:43.381963 28984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:43.382231 28986 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:42:43.384442 28986 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:42:43.384773 28986 catalog_manager.cc:1259] Loaded cluster ID: c302d5e480d84aabadda95d4bba72d12
I20260926 16:42:43.384815 28986 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:42:43.385399 28986 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:42:43.385735 28986 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be: Loaded TSK: 0
I20260926 16:42:43.386456 28986 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:42:43.456915 28440 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54946:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:42:43.460028 28440 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:42:43.462921 28982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:43.462671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34799 (local address 127.0.0.1:55124)
0926 16:42:43.462793 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:43.462847 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:34799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 16:42:43.462960 28259 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [term 3 FOLLOWER]: Refusing update from remote peer 324ff229b53c4d59b8116721df0529be: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260926 16:42:43.463455 28984 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 35885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260926 16:42:43.463649 28982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:43.463416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34799 (local address 127.0.0.1:55132)
0926 16:42:43.463517 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:43.463567 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:34799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 16:42:43.463791 28424 consensus_peers.cc:487] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be -> Peer 39e91c6cfaa7419891443717312b224d (127.21.12.126:34799): Couldn't send request to peer 39e91c6cfaa7419891443717312b224d. Status: Network error: Client connection negotiation failed: client connection to 127.21.12.126:34799: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 16:42:43.464875 28988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 324ff229b53c4d59b8116721df0529be. Latest consensus state: current_term: 3 leader_uuid: "324ff229b53c4d59b8116721df0529be" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } peers { permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 35885 } attrs { promote: false } } peers { permanent_uuid: "324ff229b53c4d59b8116721df0529be" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 42743 } attrs { promote: false } } }
I20260926 16:42:43.465029 28988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:42:43.466164 28984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "324ff229b53c4d59b8116721df0529be" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } peers { permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 35885 } attrs { promote: false } } peers { permanent_uuid: "324ff229b53c4d59b8116721df0529be" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 42743 } attrs { promote: false } } }
I20260926 16:42:43.466287 28984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:43.466413 28988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "324ff229b53c4d59b8116721df0529be" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "39e91c6cfaa7419891443717312b224d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34799 } } peers { permanent_uuid: "1f70fdb997ce4656b970b08582ad2556" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 35885 } attrs { promote: false } } peers { permanent_uuid: "324ff229b53c4d59b8116721df0529be" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 42743 } attrs { promote: false } } }
I20260926 16:42:43.466552 28988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f70fdb997ce4656b970b08582ad2556 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:42:43.476243 28997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:43.472697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32852 (local address 127.21.12.65:37069)
0926 16:42:43.472937 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:43.472942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:43.472960 (+ 18us) server_negotiation.cc:407] Connection header received
0926 16:42:43.473012 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:43.473015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:43.473061 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:43.473149 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:43.473461 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:43.474097 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:43.475280 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:43.475701 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:43.475825 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:43.475887 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:43.476060 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 16:42:43.476131 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":64,"threads_started":1}
I20260926 16:42:43.476243 28996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:43.472343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:37069 (local address 127.0.0.1:32852)
0926 16:42:43.472835 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:43.472848 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:43.472931 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:43.473184 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:43.473189 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:43.473206 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:43.473327 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:43.473333 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:43.474189 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:43.474192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:43.475157 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:43.475166 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:43.475823 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:43.475826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:43.475919 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:43.475941 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 16:42:43.476041 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 16:42:43.476108 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":98,"threads_started":1}
I20260926 16:42:43.476966 28982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:43.472162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.67:42229 (local address 127.0.0.1:37194)
0926 16:42:43.472308 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:43.472341 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:43.472434 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:43.473909 (+ 1475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:43.473912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:43.473927 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:43.474012 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:43.474018 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:43.474795 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:43.474798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:43.475822 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:43.475832 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:43.476697 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:43.476701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:43.476786 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:43.476802 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 16:42:43.476839 (+ 37us) client_negotiation.cc:240] Negotiation successful
0926 16:42:43.476884 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 16:42:43.477218 28553 tablet_service.cc:1467] Processing CreateTablet for tablet 4128e371f07b4338987022481426c4f5 (DEFAULT_TABLE table=test-table [id=665e873d23c44efa9034de378b7a6b24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:43.477452 28994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:43.472252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37194 (local address 127.21.12.67:42229)
0926 16:42:43.473637 (+ 1385us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:43.473643 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:43.473657 (+ 14us) server_negotiation.cc:407] Connection header received
0926 16:42:43.473713 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:43.473715 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:43.473803 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:43.473888 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:43.474138 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:43.474684 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:43.475946 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:43.476557 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:43.477160 (+ 603us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:43.477222 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:43.477252 (+ 30us) server_negotiation.cc:299] Negotiation successful
0926 16:42:43.477306 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":58,"threads_started":1}
I20260926 16:42:43.477527 28553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4128e371f07b4338987022481426c4f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:43.478034 28827 tablet_service.cc:1467] Processing CreateTablet for tablet 4128e371f07b4338987022481426c4f5 (DEFAULT_TABLE table=test-table [id=665e873d23c44efa9034de378b7a6b24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:43.478276 28827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4128e371f07b4338987022481426c4f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:43.480170 28999 tablet_bootstrap.cc:492] T 4128e371f07b4338987022481426c4f5 P 2d57671a949840679f2e9b9ec4ddb8de: Bootstrap starting.
I20260926 16:42:43.480962 28995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:43.472606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:41203 (local address 127.0.0.1:37500)
0926 16:42:43.474096 (+ 1490us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:43.474110 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:43.474197 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:43.476516 (+ 2319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:43.476521 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:43.476542 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:43.476629 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:43.476639 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:43.478445 (+ 1806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:43.478449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:43.479664 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:43.479673 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:43.480532 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:43.480536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:43.480637 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:43.480661 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 16:42:43.480754 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 16:42:43.480834 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1410,"thread_start_us":92,"threads_started":1}
I20260926 16:42:43.480967 28998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:43.473255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37500 (local address 127.21.12.66:41203)
0926 16:42:43.473484 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:43.473491 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:43.475279 (+ 1788us) server_negotiation.cc:407] Connection header received
0926 16:42:43.475335 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:43.475338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:43.475384 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:43.475456 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:43.477646 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:43.478302 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:43.479779 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:43.480408 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:43.480514 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:43.480595 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:43.480771 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 16:42:43.480843 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":92,"threads_started":1}
I20260926 16:42:43.481300 29000 tablet_bootstrap.cc:492] T 4128e371f07b4338987022481426c4f5 P b27e9cb575dc439eae6edd24bc55de6c: Bootstrap starting.
I20260926 16:42:43.481360 28999 tablet_bootstrap.cc:654] T 4128e371f07b4338987022481426c4f5 P 2d57671a949840679f2e9b9ec4ddb8de: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:43.481500 28690 tablet_service.cc:1467] Processing CreateTablet for tablet 4128e371f07b4338987022481426c4f5 (DEFAULT_TABLE table=test-table [id=665e873d23c44efa9034de378b7a6b24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:43.481810 28690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4128e371f07b4338987022481426c4f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:43.482223 29000 tablet_bootstrap.cc:654] T 4128e371f07b4338987022481426c4f5 P b27e9cb575dc439eae6edd24bc55de6c: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:43.482611 28999 tablet_bootstrap.cc:492] T 4128e371f07b4338987022481426c4f5 P 2d57671a949840679f2e9b9ec4ddb8de: No bootstrap required, opened a new log
I20260926 16:42:43.482707 28999 ts_tablet_manager.cc:1397] T 4128e371f07b4338987022481426c4f5 P 2d57671a949840679f2e9b9ec4ddb8de: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 16:42:43.483211 28999 raft_consensus.cc:348] T 4128e371f07b4338987022481426c4f5 P 2d57671a949840679f2e9b9ec4ddb8de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b27e9cb575dc439eae6edd24bc55de6c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42229 } } peers { permanent_uuid: "2d57671a949840679f2e9b9ec4ddb8de" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 37069 } } peers { permanent_uuid: "0df678e3047b4d9ca7e774b8040f86b4" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 41203 } }
I20260926 16:42:43.483332 28999 raft_consensus.cc:374] T 4128e371f07b4338987022481426c4f5 P 2d57671a949840679f2e9b9ec4ddb8de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:43.483366 28999 raft_consensus.cc:729] T 4128e371f07b4338987022481426c4f5 P 2d57671a949840679f2e9b9ec4ddb8de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d57671a949840679f2e9b9ec4ddb8de, State: Initialized, Role: FOLLOWER
I20260926 16:42:43.483522 28999 consensus_queue.cc:260] T 4128e371f07b4338987022481426c4f5 P 2d57671a949840679f2e9b9ec4ddb8de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b27e9cb575dc439eae6edd24bc55de6c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42229 } } peers { permanent_uuid: "2d57671a949840679f2e9b9ec4ddb8de" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 37069 } } peers { permanent_uuid: "0df678e3047b4d9ca7e774b8040f86b4" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 41203 } }
I20260926 16:42:43.483687 29000 tablet_bootstrap.cc:492] T 4128e371f07b4338987022481426c4f5 P b27e9cb575dc439eae6edd24bc55de6c: No bootstrap required, opened a new log
I20260926 16:42:43.483767 29000 ts_tablet_manager.cc:1397] T 4128e371f07b4338987022481426c4f5 P b27e9cb575dc439eae6edd24bc55de6c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260926 16:42:43.483858 28999 ts_tablet_manager.cc:1428] T 4128e371f07b4338987022481426c4f5 P 2d57671a949840679f2e9b9ec4ddb8de: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260926 16:42:43.484246 29004 tablet_bootstrap.cc:492] T 4128e371f07b4338987022481426c4f5 P 0df678e3047b4d9ca7e774b8040f86b4: Bootstrap starting.
I20260926 16:42:43.484236 29000 raft_consensus.cc:348] T 4128e371f07b4338987022481426c4f5 P b27e9cb575dc439eae6edd24bc55de6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b27e9cb575dc439eae6edd24bc55de6c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42229 } } peers { permanent_uuid: "2d57671a949840679f2e9b9ec4ddb8de" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 37069 } } peers { permanent_uuid: "0df678e3047b4d9ca7e774b8040f86b4" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 41203 } }
I20260926 16:42:43.484357 29000 raft_consensus.cc:374] T 4128e371f07b4338987022481426c4f5 P b27e9cb575dc439eae6edd24bc55de6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:43.484409 29000 raft_consensus.cc:729] T 4128e371f07b4338987022481426c4f5 P b27e9cb575dc439eae6edd24bc55de6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b27e9cb575dc439eae6edd24bc55de6c, State: Initialized, Role: FOLLOWER
I20260926 16:42:43.484536 29000 consensus_queue.cc:260] T 4128e371f07b4338987022481426c4f5 P b27e9cb575dc439eae6edd24bc55de6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b27e9cb575dc439eae6edd24bc55de6c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42229 } } peers { permanent_uuid: "2d57671a949840679f2e9b9ec4ddb8de" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 37069 } } peers { permanent_uuid: "0df678e3047b4d9ca7e774b8040f86b4" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 41203 } }
I20260926 16:42:43.484855 29000 ts_tablet_manager.cc:1428] T 4128e371f07b4338987022481426c4f5 P b27e9cb575dc439eae6edd24bc55de6c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 16:42:43.485409 29004 tablet_bootstrap.cc:654] T 4128e371f07b4338987022481426c4f5 P 0df678e3047b4d9ca7e774b8040f86b4: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:43.486246 29006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:43.485289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34799 (local address 127.21.12.67:58737)
0926 16:42:43.486099 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:43.486156 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:34799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":746,"thread_start_us":101,"threads_started":1}
I20260926 16:42:43.487016 29006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:43.486631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34799 (local address 127.21.12.67:49163)
0926 16:42:43.486890 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:43.486940 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:34799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":85,"threads_started":1}
I20260926 16:42:43.487506 28895 heartbeater.cc:503] Master 127.21.12.124:42743 was elected leader, sending a full tablet report...
I20260926 16:42:43.487923 29004 tablet_bootstrap.cc:492] T 4128e371f07b4338987022481426c4f5 P 0df678e3047b4d9ca7e774b8040f86b4: No bootstrap required, opened a new log
I20260926 16:42:43.487963 28619 heartbeater.cc:503] Master 127.21.12.124:42743 was elected leader, sending a full tablet report...
I20260926 16:42:43.488030 29004 ts_tablet_manager.cc:1397] T 4128e371f07b4338987022481426c4f5 P 0df678e3047b4d9ca7e774b8040f86b4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 16:42:43.488427 29004 raft_consensus.cc:348] T 4128e371f07b4338987022481426c4f5 P 0df678e3047b4d9ca7e774b8040f86b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b27e9cb575dc439eae6edd24bc55de6c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42229 } } peers { permanent_uuid: "2d57671a949840679f2e9b9ec4ddb8de" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 37069 } } peers { permanent_uuid: "0df678e3047b4d9ca7e774b8040f86b4" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 41203 } }
I20260926 16:42:43.488569 29004 raft_consensus.cc:374] T 4128e371f07b4338987022481426c4f5 P 0df678e3047b4d9ca7e774b8040f86b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:43.488606 29004 raft_consensus.cc:729] T 4128e371f07b4338987022481426c4f5 P 0df678e3047b4d9ca7e774b8040f86b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0df678e3047b4d9ca7e774b8040f86b4, State: Initialized, Role: FOLLOWER
I20260926 16:42:43.488761 29004 consensus_queue.cc:260] T 4128e371f07b4338987022481426c4f5 P 0df678e3047b4d9ca7e774b8040f86b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b27e9cb575dc439eae6edd24bc55de6c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42229 } } peers { permanent_uuid: "2d57671a949840679f2e9b9ec4ddb8de" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 37069 } } peers { permanent_uuid: "0df678e3047b4d9ca7e774b8040f86b4" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 41203 } }
I20260926 16:42:43.490197 29011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:43.489516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34799 (local address 127.21.12.66:54009)
0926 16:42:43.490041 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:43.490107 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:34799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":94,"threads_started":1}
I20260926 16:42:43.490501 29004 ts_tablet_manager.cc:1428] T 4128e371f07b4338987022481426c4f5 P 0df678e3047b4d9ca7e774b8040f86b4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 16:42:43.490764 28756 heartbeater.cc:503] Master 127.21.12.124:42743 was elected leader, sending a full tablet report...
I20260926 16:42:43.491102 29011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:43.490881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34799 (local address 127.21.12.66:44907)
0926 16:42:43.490972 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:43.491023 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:34799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 16:42:43.491277 29009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:43.486740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34799 (local address 127.21.12.65:46301)
0926 16:42:43.491117 (+ 4377us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:43.491174 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:34799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4299,"thread_start_us":74,"threads_started":1}
I20260926 16:42:43.492132 29009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:43.491883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34799 (local address 127.21.12.65:49145)
0926 16:42:43.492036 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:43.492084 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:34799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260926 16:42:43.493423 28898 tablet.cc:2369] T 4128e371f07b4338987022481426c4f5 P b27e9cb575dc439eae6edd24bc55de6c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:42:43.549916 28759 tablet.cc:2369] T 4128e371f07b4338987022481426c4f5 P 0df678e3047b4d9ca7e774b8040f86b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:42:43.557317 29010 raft_consensus.cc:482] T 4128e371f07b4338987022481426c4f5 P 0df678e3047b4d9ca7e774b8040f86b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:42:43.557487 29010 raft_consensus.cc:504] T 4128e371f07b4338987022481426c4f5 P 0df678e3047b4d9ca7e774b8040f86b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b27e9cb575dc439eae6edd24bc55de6c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42229 } } peers { permanent_uuid: "2d57671a949840679f2e9b9ec4ddb8de" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 37069 } } peers { permanent_uuid: "0df678e3047b4d9ca7e774b8040f86b4" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 41203 } }
I20260926 16:42:43.558022 29010 leader_election.cc:288] T 4128e371f07b4338987022481426c4f5 P 0df678e3047b4d9ca7e774b8040f86b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b27e9cb575dc439eae6edd24bc55de6c (127.21.12.67:42229), 2d57671a949840679f2e9b9ec4ddb8de (127.21.12.65:37069)
I20260926 16:42:43.558485 28573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4128e371f07b4338987022481426c4f5" candidate_uuid: "0df678e3047b4d9ca7e774b8040f86b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d57671a949840679f2e9b9ec4ddb8de" is_pre_election: true
I20260926 16:42:43.558483 28848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4128e371f07b4338987022481426c4f5" candidate_uuid: "0df678e3047b4d9ca7e774b8040f86b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b27e9cb575dc439eae6edd24bc55de6c" is_pre_election: true
I20260926 16:42:43.558636 28573 raft_consensus.cc:2454] T 4128e371f07b4338987022481426c4f5 P 2d57671a949840679f2e9b9ec4ddb8de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0df678e3047b4d9ca7e774b8040f86b4 in term 0.
I20260926 16:42:43.558636 28848 raft_consensus.cc:2454] T 4128e371f07b4338987022481426c4f5 P b27e9cb575dc439eae6edd24bc55de6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0df678e3047b4d9ca7e774b8040f86b4 in term 0.
I20260926 16:42:43.558977 28645 leader_election.cc:302] T 4128e371f07b4338987022481426c4f5 P 0df678e3047b4d9ca7e774b8040f86b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0df678e3047b4d9ca7e774b8040f86b4, b27e9cb575dc439eae6edd24bc55de6c; no voters:
I20260926 16:42:43.559190 29010 raft_consensus.cc:2781] T 4128e371f07b4338987022481426c4f5 P 0df678e3047b4d9ca7e774b8040f86b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:42:43.559266 29010 raft_consensus.cc:482] T 4128e371f07b4338987022481426c4f5 P 0df678e3047b4d9ca7e774b8040f86b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:42:43.559321 29010 raft_consensus.cc:3037] T 4128e371f07b4338987022481426c4f5 P 0df678e3047b4d9ca7e774b8040f86b4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:43.560320 29010 raft_consensus.cc:504] T 4128e371f07b4338987022481426c4f5 P 0df678e3047b4d9ca7e774b8040f86b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b27e9cb575dc439eae6edd24bc55de6c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42229 } } peers { permanent_uuid: "2d57671a949840679f2e9b9ec4ddb8de" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 37069 } } peers { permanent_uuid: "0df678e3047b4d9ca7e774b8040f86b4" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 41203 } }
I20260926 16:42:43.560696 29010 leader_election.cc:288] T 4128e371f07b4338987022481426c4f5 P 0df678e3047b4d9ca7e774b8040f86b4 [CANDIDATE]: Term 1 election: Requested vote from peers b27e9cb575dc439eae6edd24bc55de6c (127.21.12.67:42229), 2d57671a949840679f2e9b9ec4ddb8de (127.21.12.65:37069)
I20260926 16:42:43.561086 28848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4128e371f07b4338987022481426c4f5" candidate_uuid: "0df678e3047b4d9ca7e774b8040f86b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b27e9cb575dc439eae6edd24bc55de6c"
I20260926 16:42:43.561162 28573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4128e371f07b4338987022481426c4f5" candidate_uuid: "0df678e3047b4d9ca7e774b8040f86b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d57671a949840679f2e9b9ec4ddb8de"
I20260926 16:42:43.561225 28848 raft_consensus.cc:3037] T 4128e371f07b4338987022481426c4f5 P b27e9cb575dc439eae6edd24bc55de6c [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:43.561303 28573 raft_consensus.cc:3037] T 4128e371f07b4338987022481426c4f5 P 2d57671a949840679f2e9b9ec4ddb8de [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:43.562266 28848 raft_consensus.cc:2454] T 4128e371f07b4338987022481426c4f5 P b27e9cb575dc439eae6edd24bc55de6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0df678e3047b4d9ca7e774b8040f86b4 in term 1.
I20260926 16:42:43.562270 28573 raft_consensus.cc:2454] T 4128e371f07b4338987022481426c4f5 P 2d57671a949840679f2e9b9ec4ddb8de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0df678e3047b4d9ca7e774b8040f86b4 in term 1.
I20260926 16:42:43.562569 28646 leader_election.cc:302] T 4128e371f07b4338987022481426c4f5 P 0df678e3047b4d9ca7e774b8040f86b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0df678e3047b4d9ca7e774b8040f86b4, 2d57671a949840679f2e9b9ec4ddb8de; no voters:
I20260926 16:42:43.562742 29010 raft_consensus.cc:2781] T 4128e371f07b4338987022481426c4f5 P 0df678e3047b4d9ca7e774b8040f86b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:43.562947 29010 raft_consensus.cc:686] T 4128e371f07b4338987022481426c4f5 P 0df678e3047b4d9ca7e774b8040f86b4 [term 1 LEADER]: Becoming Leader. State: Replica: 0df678e3047b4d9ca7e774b8040f86b4, State: Running, Role: LEADER
I20260926 16:42:43.563099 29010 consensus_queue.cc:237] T 4128e371f07b4338987022481426c4f5 P 0df678e3047b4d9ca7e774b8040f86b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b27e9cb575dc439eae6edd24bc55de6c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42229 } } peers { permanent_uuid: "2d57671a949840679f2e9b9ec4ddb8de" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 37069 } } peers { permanent_uuid: "0df678e3047b4d9ca7e774b8040f86b4" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 41203 } }
I20260926 16:42:43.563874 29011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:43.563595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34799 (local address 127.21.12.66:58421)
0926 16:42:43.563734 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:43.563796 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:34799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 16:42:43.564994 28440 catalog_manager.cc:5495] T 4128e371f07b4338987022481426c4f5 P 0df678e3047b4d9ca7e774b8040f86b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0df678e3047b4d9ca7e774b8040f86b4 (127.21.12.66). New cstate: current_term: 1 leader_uuid: "0df678e3047b4d9ca7e774b8040f86b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b27e9cb575dc439eae6edd24bc55de6c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d57671a949840679f2e9b9ec4ddb8de" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 37069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0df678e3047b4d9ca7e774b8040f86b4" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 41203 } health_report { overall_health: HEALTHY } } }
I20260926 16:42:43.599618 28998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:43.596561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37510 (local address 127.21.12.66:41203)
0926 16:42:43.596690 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:43.596694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:43.596711 (+ 17us) server_negotiation.cc:407] Connection header received
0926 16:42:43.596768 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:43.596770 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:43.596818 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:43.596896 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:42:43.597174 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:43.597710 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:43.598597 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:43.598814 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:43.598921 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:43.599181 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:42:43.599248 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:42:43.599326 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:43.599418 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 16:42:43.599471 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 16:42:43.601608 28573 raft_consensus.cc:1261] T 4128e371f07b4338987022481426c4f5 P 2d57671a949840679f2e9b9ec4ddb8de [term 1 FOLLOWER]: Refusing update from remote peer 0df678e3047b4d9ca7e774b8040f86b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:42:43.601660 28848 raft_consensus.cc:1261] T 4128e371f07b4338987022481426c4f5 P b27e9cb575dc439eae6edd24bc55de6c [term 1 FOLLOWER]: Refusing update from remote peer 0df678e3047b4d9ca7e774b8040f86b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:42:43.602103 29012 consensus_queue.cc:1035] T 4128e371f07b4338987022481426c4f5 P 0df678e3047b4d9ca7e774b8040f86b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d57671a949840679f2e9b9ec4ddb8de" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 37069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:42:43.602264 29010 consensus_queue.cc:1035] T 4128e371f07b4338987022481426c4f5 P 0df678e3047b4d9ca7e774b8040f86b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b27e9cb575dc439eae6edd24bc55de6c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:42:43.604236 29008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:43.603647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34799 (local address 127.21.12.67:44111)
0926 16:42:43.604061 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:43.604128 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:34799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339}
I20260926 16:42:43.604550 29009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:43.603657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34799 (local address 127.21.12.65:58103)
0926 16:42:43.604411 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:43.604476 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:34799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":681}
I20260926 16:42:43.607754 29011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:43.607486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34799 (local address 127.21.12.66:34349)
0926 16:42:43.607622 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:43.607672 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:34799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 16:42:43.608428 29009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:43.608113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34799 (local address 127.21.12.65:47183)
0926 16:42:43.608286 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:43.608361 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:34799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260926 16:42:43.610735 29011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:43.610473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34799 (local address 127.21.12.66:50211)
0926 16:42:43.610611 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:43.610664 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:34799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 16:42:43.612226 29008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:43.611935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34799 (local address 127.21.12.67:60609)
0926 16:42:43.612110 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:43.612161 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:34799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260926 16:42:43.613215 28440 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:42:43.613359 28440 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:42:43.616885 28440 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 324ff229b53c4d59b8116721df0529be: Sending DeleteTablet for 3 replicas of tablet 4128e371f07b4338987022481426c4f5
I20260926 16:42:43.617609 28553 tablet_service.cc:1514] Processing DeleteTablet for tablet 4128e371f07b4338987022481426c4f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:43 UTC) from {username='oryx', principal='oryx/127.21.12.124@KRBTEST.COM'} at 127.0.0.1:32852
I20260926 16:42:43.617786 28827 tablet_service.cc:1514] Processing DeleteTablet for tablet 4128e371f07b4338987022481426c4f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:43 UTC) from {username='oryx', principal='oryx/127.21.12.124@KRBTEST.COM'} at 127.0.0.1:37194
I20260926 16:42:43.617892 28690 tablet_service.cc:1514] Processing DeleteTablet for tablet 4128e371f07b4338987022481426c4f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:43 UTC) from {username='oryx', principal='oryx/127.21.12.124@KRBTEST.COM'} at 127.0.0.1:37500
I20260926 16:42:43.618064 29025 tablet_replica.cc:330] T 4128e371f07b4338987022481426c4f5 P b27e9cb575dc439eae6edd24bc55de6c: stopping tablet replica
I20260926 16:42:43.618220 29024 tablet_replica.cc:330] T 4128e371f07b4338987022481426c4f5 P 2d57671a949840679f2e9b9ec4ddb8de: stopping tablet replica
I20260926 16:42:43.618219 29025 raft_consensus.cc:2229] T 4128e371f07b4338987022481426c4f5 P b27e9cb575dc439eae6edd24bc55de6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:42:43.618237 29026 tablet_replica.cc:330] T 4128e371f07b4338987022481426c4f5 P 0df678e3047b4d9ca7e774b8040f86b4: stopping tablet replica
I20260926 16:42:43.618341 29025 raft_consensus.cc:2258] T 4128e371f07b4338987022481426c4f5 P b27e9cb575dc439eae6edd24bc55de6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:42:43.618357 29024 raft_consensus.cc:2229] T 4128e371f07b4338987022481426c4f5 P 2d57671a949840679f2e9b9ec4ddb8de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:42:43.618446 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 28402
I20260926 16:42:43.618546 29026 raft_consensus.cc:2229] T 4128e371f07b4338987022481426c4f5 P 0df678e3047b4d9ca7e774b8040f86b4 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:42:43.618747 29026 raft_consensus.cc:2258] T 4128e371f07b4338987022481426c4f5 P 0df678e3047b4d9ca7e774b8040f86b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:42:43.620364 29026 ts_tablet_manager.cc:1905] T 4128e371f07b4338987022481426c4f5 P 0df678e3047b4d9ca7e774b8040f86b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:42:43.622346 29025 ts_tablet_manager.cc:1905] T 4128e371f07b4338987022481426c4f5 P b27e9cb575dc439eae6edd24bc55de6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:42:43.622922 29026 ts_tablet_manager.cc:1918] T 4128e371f07b4338987022481426c4f5 P 0df678e3047b4d9ca7e774b8040f86b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:42:43.622996 29026 log.cc:1196] T 4128e371f07b4338987022481426c4f5 P 0df678e3047b4d9ca7e774b8040f86b4: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-1/wal/wals/4128e371f07b4338987022481426c4f5
I20260926 16:42:43.623317 29026 ts_tablet_manager.cc:1939] T 4128e371f07b4338987022481426c4f5 P 0df678e3047b4d9ca7e774b8040f86b4: Deleting consensus metadata
I20260926 16:42:43.623997 28425 catalog_manager.cc:4841] TS 0df678e3047b4d9ca7e774b8040f86b4 (127.21.12.66:41203): tablet 4128e371f07b4338987022481426c4f5 (table test-table [id=665e873d23c44efa9034de378b7a6b24]) successfully deleted
I20260926 16:42:43.624821 29025 ts_tablet_manager.cc:1918] T 4128e371f07b4338987022481426c4f5 P b27e9cb575dc439eae6edd24bc55de6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:42:43.624882 29025 log.cc:1196] T 4128e371f07b4338987022481426c4f5 P b27e9cb575dc439eae6edd24bc55de6c: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440924937351-21553-0/minicluster-data/ts-2/wal/wals/4128e371f07b4338987022481426c4f5
I20260926 16:42:43.625146 29025 ts_tablet_manager.cc:1939] T 4128e371f07b4338987022481426c4f5 P b27e9cb575dc439eae6edd24bc55de6c: Deleting consensus metadata
I20260926 16:42:43.625787 28426 catalog_manager.cc:4841] TS b27e9cb575dc439eae6edd24bc55de6c (127.21.12.67:42229): tablet 4128e371f07b4338987022481426c4f5 (table test-table [id=665e873d23c44efa9034de378b7a6b24]) successfully deleted
I20260926 16:42:43.625931 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 28627
I20260926 16:42:43.626180 28995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:43.625865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:37069 (local address 127.0.0.1:32860)
0926 16:42:43.626042 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:43.626097 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.65:37069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260926 16:42:43.629906 28427 catalog_manager.cc:4584] TS 2d57671a949840679f2e9b9ec4ddb8de (127.21.12.65:37069): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4128e371f07b4338987022481426c4f5: Network error: Client connection negotiation failed: client connection to 127.21.12.65:37069: connect: Connection refused (error 111)
I20260926 16:42:43.632934 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 28763
I20260926 16:42:43.638532 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 28120
I20260926 16:42:43.644228 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 28221
2026-09-26T16:42:43Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7209 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:42:43 dist-test-slave-n1hw krb5kdc[29029](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:42:43 dist-test-slave-n1hw krb5kdc[29029](info): set up 2 sockets
Sep 26 16:42:43 dist-test-slave-n1hw krb5kdc[29029](info): commencing operation
krb5kdc: starting...
W20260926 16:42:44.688620 21553 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Sep 26 16:42:44 dist-test-slave-n1hw krb5kdc[29029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:42:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:42:44Z Disabled control of system clock
WARNING: no policy specified for oryx/127.21.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.126.keytab.
Entry for principal oryx/127.21.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.126.keytab.
I20260926 16:42:44.808290 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:33749
--webserver_interface=127.21.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.126.keytab
--principal=oryx/127.21.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:34035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:33749
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:44.917631 29042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:44.917899 29042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:44.917939 29042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:44.921976 29042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:42:44.922034 29042 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 16:42:44.922050 29042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:44.922068 29042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:44.922084 29042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:42:44.922099 29042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:42:44.927244 29042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:34035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:33749
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.126.keytab
--principal=oryx/127.21.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:33749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:44.928184 29042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:44.929131 29042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:44.937433 29050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:44.937462 29048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:44.937459 29047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:44.937513 29042 server_base.cc:1034] running on GCE node
I20260926 16:42:44.938213 29042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:44.939258 29042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:44.940452 29042 hybrid_clock.cc:648] HybridClock initialized: now 1790440964940438 us; error 25 us; skew 500 ppm
Sep 26 16:42:44 dist-test-slave-n1hw krb5kdc[29029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440964, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:44.946439 29042 init.cc:377] Logged in from keytab as oryx/127.21.12.126@KRBTEST.COM (short username oryx)
I20260926 16:42:44.947322 29042 webserver.cc:460] Webserver started at http://127.21.12.126:36235/ using document root <none> and password file <none>
I20260926 16:42:44.947892 29042 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:44.947988 29042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:44.948241 29042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:44.950356 29042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "2a2017701436402f9ebb08a9813ae014"
format_stamp: "Formatted at 2026-09-26 16:42:44 on dist-test-slave-n1hw"
server_key: "7c348fe210d66090d246a6a2ed1cb1af"
server_key_iv: "9af1138b276d1be1abf0328d3b7173e0"
server_key_version: "encryptionkey@0"
I20260926 16:42:44.950917 29042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "2a2017701436402f9ebb08a9813ae014"
format_stamp: "Formatted at 2026-09-26 16:42:44 on dist-test-slave-n1hw"
server_key: "7c348fe210d66090d246a6a2ed1cb1af"
server_key_iv: "9af1138b276d1be1abf0328d3b7173e0"
server_key_version: "encryptionkey@0"
I20260926 16:42:44.954689 29042 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:42:44.956986 29057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:44.958271 29042 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:42:44.958425 29042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "2a2017701436402f9ebb08a9813ae014"
format_stamp: "Formatted at 2026-09-26 16:42:44 on dist-test-slave-n1hw"
server_key: "7c348fe210d66090d246a6a2ed1cb1af"
server_key_iv: "9af1138b276d1be1abf0328d3b7173e0"
server_key_version: "encryptionkey@0"
I20260926 16:42:44.958539 29042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:44.971130 29042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:44.974109 29042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:44.974308 29042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:44.982357 29042 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.126:33749
I20260926 16:42:44.982370 29119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.126:33749 every 8 connection(s)
I20260926 16:42:44.983269 29042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:42:44.984283 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 29042
I20260926 16:42:44.984419 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:42:44.984697 21553 external_mini_cluster.cc:1428] Setting key 561ea5c83afc4abaf86c8c88c7369b85
I20260926 16:42:44.986238 29120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:44.992911 29120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a2017701436402f9ebb08a9813ae014: Bootstrap starting.
Sep 26 16:42:44 dist-test-slave-n1hw krb5kdc[29029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM
I20260926 16:42:44.995543 29120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a2017701436402f9ebb08a9813ae014: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:44.996359 29120 log.cc:824] T 00000000000000000000000000000000 P 2a2017701436402f9ebb08a9813ae014: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:44.998548 29120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a2017701436402f9ebb08a9813ae014: No bootstrap required, opened a new log
I20260926 16:42:45.000437 29123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:44.986115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47194 (local address 127.21.12.126:33749)
0926 16:42:44.986568 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:44.986588 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:44.986630 (+ 42us) server_negotiation.cc:407] Connection header received
0926 16:42:44.987437 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:44.987459 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:44.987866 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:44.988226 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:44.988886 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:44.990398 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:44.991221 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:44.991920 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:44.992074 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:44.994876 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:44.994881 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:44.994893 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:44.994919 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:44.997583 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:44.998107 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:44.998112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:44.998116 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:44.998223 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:44.998521 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:44.998523 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:44.998524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:44.998940 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:44.999081 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:44.999415 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 16:42:44.999735 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":140,"threads_started":1}
I20260926 16:42:45.001739 29120 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a2017701436402f9ebb08a9813ae014 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a2017701436402f9ebb08a9813ae014" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 33749 } }
I20260926 16:42:45.002022 29120 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a2017701436402f9ebb08a9813ae014 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:45.002115 29120 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a2017701436402f9ebb08a9813ae014 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a2017701436402f9ebb08a9813ae014, State: Initialized, Role: FOLLOWER
I20260926 16:42:45.002585 29120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a2017701436402f9ebb08a9813ae014 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a2017701436402f9ebb08a9813ae014" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 33749 } }
I20260926 16:42:45.002731 29120 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a2017701436402f9ebb08a9813ae014 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:42:45.002799 29120 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a2017701436402f9ebb08a9813ae014 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:42:45.002895 29120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a2017701436402f9ebb08a9813ae014 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:45.003780 29120 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a2017701436402f9ebb08a9813ae014 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a2017701436402f9ebb08a9813ae014" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 33749 } }
I20260926 16:42:45.004110 29120 leader_election.cc:302] T 00000000000000000000000000000000 P 2a2017701436402f9ebb08a9813ae014 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a2017701436402f9ebb08a9813ae014; no voters:
I20260926 16:42:45.004401 29120 leader_election.cc:288] T 00000000000000000000000000000000 P 2a2017701436402f9ebb08a9813ae014 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:42:45.004518 29125 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a2017701436402f9ebb08a9813ae014 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:45.004810 29125 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a2017701436402f9ebb08a9813ae014 [term 1 LEADER]: Becoming Leader. State: Replica: 2a2017701436402f9ebb08a9813ae014, State: Running, Role: LEADER
I20260926 16:42:45.005097 29125 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a2017701436402f9ebb08a9813ae014 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a2017701436402f9ebb08a9813ae014" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 33749 } }
I20260926 16:42:45.005541 29120 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a2017701436402f9ebb08a9813ae014 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:42:45.006686 29127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a2017701436402f9ebb08a9813ae014 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a2017701436402f9ebb08a9813ae014. Latest consensus state: current_term: 1 leader_uuid: "2a2017701436402f9ebb08a9813ae014" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a2017701436402f9ebb08a9813ae014" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 33749 } } }
I20260926 16:42:45.006819 29127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a2017701436402f9ebb08a9813ae014 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:45.006925 29126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a2017701436402f9ebb08a9813ae014 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a2017701436402f9ebb08a9813ae014" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a2017701436402f9ebb08a9813ae014" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 33749 } } }
I20260926 16:42:45.007022 29126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a2017701436402f9ebb08a9813ae014 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:45.007238 29134 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:42:45.010116 29134 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:42:45.014968 29134 catalog_manager.cc:1347] Generated new cluster ID: e16e4cac529c4ffa8398a55b53ccaf57
I20260926 16:42:45.015029 29134 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:42:45.039642 29134 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:42:45.040894 29134 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:42:45.046388 29134 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a2017701436402f9ebb08a9813ae014: Generated new TSK 0
I20260926 16:42:45.047201 29134 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.21.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.65.keytab.
Entry for principal oryx/127.21.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.65.keytab.
I20260926 16:42:45.077582 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.65:0
--local_ip_for_outbound_sockets=127.21.12.65
--webserver_interface=127.21.12.65
--webserver_port=0
--tserver_master_addrs=127.21.12.126:33749
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.65.keytab
--principal=oryx/127.21.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:34035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:45.187007 29146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:45.187268 29146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:45.187311 29146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:45.191349 29146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:45.191407 29146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:45.191445 29146 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 16:42:45.191505 29146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.65
I20260926 16:42:45.196699 29146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:34035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.65.keytab
--principal=oryx/127.21.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.12.126:33749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.65
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:45.197721 29146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:45.198822 29146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:45.208420 29154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:45.208452 29151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:45.208429 29152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:45.208753 29146 server_base.cc:1034] running on GCE node
I20260926 16:42:45.209146 29146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:45.209849 29146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:45.211007 29146 hybrid_clock.cc:648] HybridClock initialized: now 1790440965210995 us; error 23 us; skew 500 ppm
Sep 26 16:42:45 dist-test-slave-n1hw krb5kdc[29029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440965, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:45.213657 29146 init.cc:377] Logged in from keytab as oryx/127.21.12.65@KRBTEST.COM (short username oryx)
I20260926 16:42:45.214507 29146 webserver.cc:460] Webserver started at http://127.21.12.65:37955/ using document root <none> and password file <none>
I20260926 16:42:45.215018 29146 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:45.215087 29146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:45.215246 29146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:45.217203 29146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-0/data/instance:
uuid: "a870f8287ca3464b8746958bd1fe700f"
format_stamp: "Formatted at 2026-09-26 16:42:45 on dist-test-slave-n1hw"
server_key: "acc99ebbb5b4d0f5608de7c40d1b7e88"
server_key_iv: "8cae4c67c94f2e184f5e23dca1ae0e22"
server_key_version: "encryptionkey@0"
I20260926 16:42:45.217697 29146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance:
uuid: "a870f8287ca3464b8746958bd1fe700f"
format_stamp: "Formatted at 2026-09-26 16:42:45 on dist-test-slave-n1hw"
server_key: "acc99ebbb5b4d0f5608de7c40d1b7e88"
server_key_iv: "8cae4c67c94f2e184f5e23dca1ae0e22"
server_key_version: "encryptionkey@0"
I20260926 16:42:45.221475 29146 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 16:42:45.224012 29161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:45.225252 29146 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:42:45.225347 29146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-0/wal
uuid: "a870f8287ca3464b8746958bd1fe700f"
format_stamp: "Formatted at 2026-09-26 16:42:45 on dist-test-slave-n1hw"
server_key: "acc99ebbb5b4d0f5608de7c40d1b7e88"
server_key_iv: "8cae4c67c94f2e184f5e23dca1ae0e22"
server_key_version: "encryptionkey@0"
I20260926 16:42:45.225431 29146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:45.240522 29146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:45.243774 29146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:45.243922 29146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:45.245235 29146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:45.245314 29146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:45.245393 29146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:45.245438 29146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 16:42:45 dist-test-slave-n1hw krb5kdc[29029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440965, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.65@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM
I20260926 16:42:45.257011 29146 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.65:38091
I20260926 16:42:45.257107 29276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.65:38091 every 8 connection(s)
I20260926 16:42:45.258157 29146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
I20260926 16:42:45.260710 29123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.246227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:60161 (local address 127.21.12.126:33749)
0926 16:42:45.246359 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:45.246363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:45.247383 (+ 1020us) server_negotiation.cc:407] Connection header received
0926 16:42:45.248302 (+ 919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:45.248305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:45.248364 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:45.248478 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:45.250082 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.250989 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.252765 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.253190 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.253323 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:45.256716 (+ 3393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:45.256722 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:45.256725 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:45.256760 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:45.258618 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:45.259227 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:45.259232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:45.259234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:45.259380 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:45.259692 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:45.259695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:45.259696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:45.259896 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:45.259988 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:45.260455 (+ 467us) server_negotiation.cc:299] Negotiation successful
0926 16:42:45.260576 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 16:42:45.261529 29170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.246748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:33749 (local address 127.21.12.65:60161)
0926 16:42:45.247222 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:45.247268 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:45.248097 (+ 829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:45.248692 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:45.248701 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:45.249145 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:45.249901 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:45.249925 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.251137 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.251141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:45.252624 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:45.252636 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.253370 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.253374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:45.253591 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:45.254336 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:45.254355 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:45.256497 (+ 2142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:45.258772 (+ 2275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:45.258778 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:45.258792 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:45.259113 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:45.259497 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:45.259499 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:45.259501 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:45.259625 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:45.259991 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:45.259995 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:45.260218 (+ 223us) client_negotiation.cc:769] Sending connection context
0926 16:42:45.260366 (+ 148us) client_negotiation.cc:240] Negotiation successful
0926 16:42:45.260599 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":124,"threads_started":1}
I20260926 16:42:45.262814 29277 heartbeater.cc:347] Connected to a master server at 127.21.12.126:33749
I20260926 16:42:45.262970 29277 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:45.263432 29277 heartbeater.cc:511] Master 127.21.12.126:33749 requested a full tablet report, sending...
I20260926 16:42:45.263872 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 29146
I20260926 16:42:45.264061 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance
I20260926 16:42:45.264437 21553 external_mini_cluster.cc:1428] Setting key 86e3b4919f9efadf4aa7cdee273154a2
I20260926 16:42:45.265897 29070 ts_manager.cc:194] Registered new tserver with Master: a870f8287ca3464b8746958bd1fe700f (127.21.12.65:38091)
I20260926 16:42:45.267242 29070 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.12.65@KRBTEST.COM'} at 127.21.12.65:60161
I20260926 16:42:45.274863 29283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.265374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:33749 (local address 127.21.12.65:59063)
0926 16:42:45.265704 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:45.265720 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:45.265850 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:45.266235 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:45.266238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:45.266439 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:45.266559 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:45.266567 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.267835 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.267842 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:45.268713 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:45.268724 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.269502 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.269655 (+ 153us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:45.269743 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:45.270379 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:45.270391 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:45.270765 (+ 374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:45.273158 (+ 2393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:45.273161 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:45.273162 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:45.273364 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:45.273727 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:45.273729 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:45.273731 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:45.273877 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:45.274419 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:45.274421 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:45.274505 (+ 84us) client_negotiation.cc:769] Sending connection context
0926 16:42:45.274596 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 16:42:45.274710 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":145,"threads_started":1}
I20260926 16:42:45.275279 29123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.265708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:59063 (local address 127.21.12.126:33749)
0926 16:42:45.265983 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:45.265988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:45.266004 (+ 16us) server_negotiation.cc:407] Connection header received
0926 16:42:45.266048 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:45.266053 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:45.266106 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:45.266204 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:45.266705 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.267643 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.268875 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.269348 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.269491 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:45.270936 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:45.270944 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:45.270951 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:45.270989 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:45.272994 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:45.273477 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:45.273484 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:45.273488 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:45.273609 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:45.273987 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:45.273992 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:45.273996 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:45.274289 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:45.274420 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:45.274723 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 16:42:45.274875 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188}
WARNING: no policy specified for oryx/127.21.12.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.66.keytab.
Entry for principal oryx/127.21.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.66.keytab.
I20260926 16:42:45.298686 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.66:0
--local_ip_for_outbound_sockets=127.21.12.66
--webserver_interface=127.21.12.66
--webserver_port=0
--tserver_master_addrs=127.21.12.126:33749
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.66.keytab
--principal=oryx/127.21.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:34035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:45.412602 29286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:45.412900 29286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:45.412978 29286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:45.417028 29286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:45.417130 29286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:45.417203 29286 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 16:42:45.417289 29286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.66
I20260926 16:42:45.422860 29286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:34035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.66.keytab
--principal=oryx/127.21.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.12.126:33749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.66
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:45.424008 29286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:45.425051 29286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:45.434726 29292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:45.434717 29294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:45.434717 29291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:45.435108 29286 server_base.cc:1034] running on GCE node
I20260926 16:42:45.435490 29286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:45.436164 29286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:45.437359 29286 hybrid_clock.cc:648] HybridClock initialized: now 1790440965437323 us; error 50 us; skew 500 ppm
Sep 26 16:42:45 dist-test-slave-n1hw krb5kdc[29029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440965, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:45.440210 29286 init.cc:377] Logged in from keytab as oryx/127.21.12.66@KRBTEST.COM (short username oryx)
I20260926 16:42:45.441087 29286 webserver.cc:460] Webserver started at http://127.21.12.66:33463/ using document root <none> and password file <none>
I20260926 16:42:45.441644 29286 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:45.441730 29286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:45.442013 29286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:45.444006 29286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-1/data/instance:
uuid: "1abc5ef5f6924b23a687a4aa4df47af6"
format_stamp: "Formatted at 2026-09-26 16:42:45 on dist-test-slave-n1hw"
server_key: "5e153d2944dd025b1a32a006b7c4a1a0"
server_key_iv: "cc7b4c6405648531b8d2c869e6e7123c"
server_key_version: "encryptionkey@0"
I20260926 16:42:45.444561 29286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance:
uuid: "1abc5ef5f6924b23a687a4aa4df47af6"
format_stamp: "Formatted at 2026-09-26 16:42:45 on dist-test-slave-n1hw"
server_key: "5e153d2944dd025b1a32a006b7c4a1a0"
server_key_iv: "cc7b4c6405648531b8d2c869e6e7123c"
server_key_version: "encryptionkey@0"
I20260926 16:42:45.448292 29286 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260926 16:42:45.450802 29301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:45.452139 29286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:42:45.452271 29286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-1/wal
uuid: "1abc5ef5f6924b23a687a4aa4df47af6"
format_stamp: "Formatted at 2026-09-26 16:42:45 on dist-test-slave-n1hw"
server_key: "5e153d2944dd025b1a32a006b7c4a1a0"
server_key_iv: "cc7b4c6405648531b8d2c869e6e7123c"
server_key_version: "encryptionkey@0"
I20260926 16:42:45.452382 29286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:45.467094 29286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:45.470379 29286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:45.470592 29286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:45.472054 29286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:45.472126 29286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:45.472195 29286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:45.472239 29286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 16:42:45 dist-test-slave-n1hw krb5kdc[29029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440965, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.66@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM
I20260926 16:42:45.482507 29286 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.66:37159
I20260926 16:42:45.482630 29416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.66:37159 every 8 connection(s)
I20260926 16:42:45.483423 29286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
I20260926 16:42:45.485628 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 29286
I20260926 16:42:45.485805 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance
I20260926 16:42:45.486123 21553 external_mini_cluster.cc:1428] Setting key 743f17036ef7287130188a2c9dee8b8a
I20260926 16:42:45.487005 29123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.472827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:58825 (local address 127.21.12.126:33749)
0926 16:42:45.473000 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:45.473007 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:45.473827 (+ 820us) server_negotiation.cc:407] Connection header received
0926 16:42:45.474791 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:45.474794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:45.474848 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:45.474927 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:45.476653 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.477551 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.479491 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.479758 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.479842 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:45.483055 (+ 3213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:45.483058 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:45.483061 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:45.483095 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:45.484714 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:45.485190 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:45.485192 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:45.485194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:45.485267 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:45.485706 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:45.485709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:45.485717 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:45.486000 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:45.486083 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:45.486728 (+ 645us) server_negotiation.cc:299] Negotiation successful
0926 16:42:45.486860 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 16:42:45.487800 29310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.473214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:33749 (local address 127.21.12.66:58825)
0926 16:42:45.473656 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:45.473705 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:45.474582 (+ 877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:45.475292 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:45.475301 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:45.475800 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:45.476453 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:45.476476 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.477698 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.477702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:45.479333 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:45.479345 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.479937 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.479942 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:45.480153 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:45.480859 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:45.480887 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:45.482865 (+ 1978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:45.484821 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:45.484825 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:45.484836 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:45.485085 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:45.485372 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:45.485377 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:45.485382 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:45.485572 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:45.486141 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:45.486147 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:45.486444 (+ 297us) client_negotiation.cc:769] Sending connection context
0926 16:42:45.486628 (+ 184us) client_negotiation.cc:240] Negotiation successful
0926 16:42:45.486871 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":120,"threads_started":1}
I20260926 16:42:45.489137 29417 heartbeater.cc:347] Connected to a master server at 127.21.12.126:33749
I20260926 16:42:45.489286 29417 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:45.489931 29417 heartbeater.cc:511] Master 127.21.12.126:33749 requested a full tablet report, sending...
I20260926 16:42:45.491031 29070 ts_manager.cc:194] Registered new tserver with Master: 1abc5ef5f6924b23a687a4aa4df47af6 (127.21.12.66:37159)
I20260926 16:42:45.491880 29070 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.12.66@KRBTEST.COM'} at 127.21.12.66:58825
WARNING: no policy specified for oryx/127.21.12.67@KRBTEST.COM; defaulting to no policy
I20260926 16:42:45.499172 29424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.491434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:33749 (local address 127.21.12.66:45745)
0926 16:42:45.491744 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:45.491758 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:45.491881 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:45.492267 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:45.492270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:45.492479 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:45.492588 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:45.492593 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.493322 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.493325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:45.493975 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:45.493984 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.494648 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.494651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:45.494722 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:45.495250 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:45.495262 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:45.495567 (+ 305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:45.497551 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:45.497554 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:45.497555 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:45.497842 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:45.498160 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:45.498162 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:45.498163 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:45.498236 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:45.498729 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:45.498735 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:45.498827 (+ 92us) client_negotiation.cc:769] Sending connection context
0926 16:42:45.498910 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 16:42:45.499013 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":91,"threads_started":1}
I20260926 16:42:45.499220 29123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.491508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:45745 (local address 127.21.12.126:33749)
0926 16:42:45.491644 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:45.491650 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:45.491820 (+ 170us) server_negotiation.cc:407] Connection header received
0926 16:42:45.492042 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:45.492044 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:45.492091 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:45.492139 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:45.492702 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.493197 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.494125 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.494516 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.494636 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:45.495724 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:45.495731 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:45.495736 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:45.495768 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:45.497444 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:45.497938 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:45.497942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:45.497947 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:45.498055 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:45.498322 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:45.498327 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:45.498331 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:45.498605 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:45.498727 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:45.499001 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 16:42:45.499112 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.67.keytab.
Entry for principal oryx/127.21.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.67.keytab.
I20260926 16:42:45.517524 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.67:0
--local_ip_for_outbound_sockets=127.21.12.67
--webserver_interface=127.21.12.67
--webserver_port=0
--tserver_master_addrs=127.21.12.126:33749
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.67.keytab
--principal=oryx/127.21.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:34035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:45.627164 29426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:45.627413 29426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:45.627486 29426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:45.631922 29426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:45.631976 29426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:45.632015 29426 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 16:42:45.632107 29426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.67
I20260926 16:42:45.637234 29426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:34035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.67.keytab
--principal=oryx/127.21.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.12.126:33749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.67
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:45.638305 29426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:45.639305 29426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:45.648778 29432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:45.648782 29431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:45.648798 29434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:45.649060 29426 server_base.cc:1034] running on GCE node
I20260926 16:42:45.649542 29426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:45.650187 29426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:45.651347 29426 hybrid_clock.cc:648] HybridClock initialized: now 1790440965651331 us; error 27 us; skew 500 ppm
Sep 26 16:42:45 dist-test-slave-n1hw krb5kdc[29029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440965, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:45.654155 29426 init.cc:377] Logged in from keytab as oryx/127.21.12.67@KRBTEST.COM (short username oryx)
I20260926 16:42:45.655118 29426 webserver.cc:460] Webserver started at http://127.21.12.67:45513/ using document root <none> and password file <none>
I20260926 16:42:45.655687 29426 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:45.655776 29426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:45.656044 29426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:45.658170 29426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-2/data/instance:
uuid: "d35b2f782ab34e35b9e401ed869c808c"
format_stamp: "Formatted at 2026-09-26 16:42:45 on dist-test-slave-n1hw"
server_key: "4f9a600edc34dafeae73fdc16762b297"
server_key_iv: "effcfb2e50dec1da386af10f93c0906d"
server_key_version: "encryptionkey@0"
I20260926 16:42:45.658730 29426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance:
uuid: "d35b2f782ab34e35b9e401ed869c808c"
format_stamp: "Formatted at 2026-09-26 16:42:45 on dist-test-slave-n1hw"
server_key: "4f9a600edc34dafeae73fdc16762b297"
server_key_iv: "effcfb2e50dec1da386af10f93c0906d"
server_key_version: "encryptionkey@0"
I20260926 16:42:45.662611 29426 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:42:45.665112 29441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:45.666397 29426 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:42:45.666523 29426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-2/wal
uuid: "d35b2f782ab34e35b9e401ed869c808c"
format_stamp: "Formatted at 2026-09-26 16:42:45 on dist-test-slave-n1hw"
server_key: "4f9a600edc34dafeae73fdc16762b297"
server_key_iv: "effcfb2e50dec1da386af10f93c0906d"
server_key_version: "encryptionkey@0"
I20260926 16:42:45.666636 29426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:45.694016 29426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:45.696898 29426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:45.697106 29426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:45.698561 29426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:45.698647 29426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:45.698740 29426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:45.698777 29426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 16:42:45 dist-test-slave-n1hw krb5kdc[29029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440965, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.67@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM
I20260926 16:42:45.709510 29426 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.67:42921
I20260926 16:42:45.709604 29556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.67:42921 every 8 connection(s)
I20260926 16:42:45.710407 29426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
I20260926 16:42:45.712586 29123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.699289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:35555 (local address 127.21.12.126:33749)
0926 16:42:45.699422 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:45.699428 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:45.700433 (+ 1005us) server_negotiation.cc:407] Connection header received
0926 16:42:45.701585 (+ 1152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:45.701590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:45.701655 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:45.701722 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:45.703044 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.703764 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.704926 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.705221 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.705365 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:45.708948 (+ 3583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:45.708951 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:45.708953 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:45.708977 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:45.710760 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:45.711337 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:45.711340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:45.711342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:45.711432 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:45.711716 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:45.711718 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:45.711719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:45.711881 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:45.711953 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:45.712380 (+ 427us) server_negotiation.cc:299] Negotiation successful
0926 16:42:45.712472 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 16:42:45.713415 29450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.699808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:33749 (local address 127.21.12.67:35555)
0926 16:42:45.700286 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:45.700327 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:45.701317 (+ 990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:45.702127 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:45.702134 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:45.702445 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:45.702896 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:45.702908 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.703896 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.703900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:45.704792 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:45.704802 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.705412 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.705417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:45.705579 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:45.706208 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:45.706247 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:45.708781 (+ 2534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:45.710893 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:45.710898 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:45.710912 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:45.711227 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:45.711523 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:45.711525 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:45.711526 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:45.711654 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:45.711966 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:45.711971 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:45.712178 (+ 207us) client_negotiation.cc:769] Sending connection context
0926 16:42:45.712328 (+ 150us) client_negotiation.cc:240] Negotiation successful
0926 16:42:45.712598 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":138,"threads_started":1}
I20260926 16:42:45.714046 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 29426
I20260926 16:42:45.714185 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance
I20260926 16:42:45.714509 21553 external_mini_cluster.cc:1428] Setting key 65b04a24f61ef0d48459d7eb4d4898bd
I20260926 16:42:45.714756 29557 heartbeater.cc:347] Connected to a master server at 127.21.12.126:33749
I20260926 16:42:45.714886 29557 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:45.715427 29557 heartbeater.cc:511] Master 127.21.12.126:33749 requested a full tablet report, sending...
I20260926 16:42:45.716447 29070 ts_manager.cc:194] Registered new tserver with Master: d35b2f782ab34e35b9e401ed869c808c (127.21.12.67:42921)
I20260926 16:42:45.717163 29070 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.12.67@KRBTEST.COM'} at 127.21.12.67:35555
I20260926 16:42:45.723695 29563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.716817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:33749 (local address 127.21.12.67:42187)
0926 16:42:45.717067 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:45.717082 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:45.717184 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:45.717458 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:45.717461 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:45.717687 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:45.717815 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:45.717825 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.718808 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.718811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:45.719393 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:45.719401 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.719852 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.719854 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:45.719946 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:45.720448 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:45.720468 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:45.720807 (+ 339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:45.722344 (+ 1537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:45.722346 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:45.722348 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:45.722552 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:45.722845 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:45.722848 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:45.722849 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:45.722931 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:45.723262 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:45.723264 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:45.723337 (+ 73us) client_negotiation.cc:769] Sending connection context
0926 16:42:45.723419 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 16:42:45.723521 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":104,"threads_started":1}
I20260926 16:42:45.723758 29123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.716872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:42187 (local address 127.21.12.126:33749)
0926 16:42:45.716997 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:45.717001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:45.717147 (+ 146us) server_negotiation.cc:407] Connection header received
0926 16:42:45.717301 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:45.717304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:45.717355 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:45.717442 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:45.717917 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.718663 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.719516 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.719750 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.719839 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:45.720925 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:45.720927 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:45.720929 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:45.720949 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:45.722187 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:45.722659 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:45.722662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:45.722663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:45.722731 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:45.723020 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:45.723022 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:45.723023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:45.723171 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:45.723246 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:45.723510 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 16:42:45.723655 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 16:42:45.728178 21553 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 16:42:45 dist-test-slave-n1hw krb5kdc[29029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440965, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 16:42:45 dist-test-slave-n1hw krb5kdc[29029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:42:45 dist-test-slave-n1hw krb5kdc[29029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
W20260926 16:42:45.751951 29123 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:45.745349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47200 (local address 127.21.12.126:33749)
0926 16:42:45.745479 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:45.745482 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:45.745494 (+ 12us) server_negotiation.cc:407] Connection header received
0926 16:42:45.745533 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:45.745535 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:45.745572 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:45.745670 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:45.746395 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.746918 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.747785 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.748007 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.748125 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:45.751786 (+ 3661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47200: BlockingRecv error: recv got EOF from 127.0.0.1:47200 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 26 16:42:45 dist-test-slave-n1hw krb5kdc[29029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440965, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM
I20260926 16:42:45.761786 29123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.753460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47216 (local address 127.21.12.126:33749)
0926 16:42:45.753639 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:45.753646 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:45.753736 (+ 90us) server_negotiation.cc:407] Connection header received
0926 16:42:45.753932 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:45.753936 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:45.753983 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:45.754063 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:45.754661 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.755157 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.756005 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.756246 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.756353 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:45.758617 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:45.758621 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:45.758624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:45.758649 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:45.760089 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:45.760521 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:45.760525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:45.760527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:45.760618 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:45.760891 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:45.760895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:45.760896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:45.761075 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:45.761218 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:45.761438 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 16:42:45.761578 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260926 16:42:45.764685 29070 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47216:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:42:45.767694 29070 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:42:45.781806 29579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.775868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:38091 (local address 127.0.0.1:58730)
0926 16:42:45.776630 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:45.776684 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:45.776852 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:45.777614 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:45.777620 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:45.777657 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:45.777946 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:45.777954 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.779387 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.779390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:45.780200 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:45.780209 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.781222 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.781229 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:45.781345 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:45.781382 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 16:42:45.781498 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 16:42:45.781614 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":41,"spinlock_wait_cycles":124928,"thread_start_us":69,"threads_started":1}
I20260926 16:42:45.781831 29581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.776017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58730 (local address 127.21.12.65:38091)
0926 16:42:45.776734 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:45.776741 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:45.776765 (+ 24us) server_negotiation.cc:407] Connection header received
0926 16:42:45.777075 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:45.777084 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:45.777304 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:45.777446 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:45.778074 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.779232 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.780328 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.781085 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.781195 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:45.781337 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:45.781543 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 16:42:45.781648 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":107,"threads_started":1}
I20260926 16:42:45.782303 29582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.776075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33050 (local address 127.21.12.67:42921)
0926 16:42:45.776533 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:45.776546 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:45.777011 (+ 465us) server_negotiation.cc:407] Connection header received
0926 16:42:45.777098 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:45.777106 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:45.777318 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:45.777464 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:45.778059 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.778878 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.780419 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.781176 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.781273 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:45.781426 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:45.781871 (+ 445us) server_negotiation.cc:299] Negotiation successful
0926 16:42:45.782197 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":92,"threads_started":1}
I20260926 16:42:45.782469 29583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.776574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:37159 (local address 127.0.0.1:33344)
0926 16:42:45.777030 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:45.777043 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:45.777126 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:45.777835 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:45.777838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:45.777853 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:45.778026 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:45.778034 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.779354 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.779357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:45.780848 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:45.780858 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.782112 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.782115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:45.782212 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:45.782235 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 16:42:45.782285 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 16:42:45.782347 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"mutex_wait_us":47,"thread_start_us":96,"threads_started":1}
I20260926 16:42:45.782639 29580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.775874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.67:42921 (local address 127.0.0.1:33050)
0926 16:42:45.776631 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:45.776684 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:45.776849 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:45.777622 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:45.777626 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:45.777661 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:45.777885 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:45.777892 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.779010 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.779013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:45.780290 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:45.780299 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.781488 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.781494 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:45.781589 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:45.781609 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 16:42:45.781673 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 16:42:45.781729 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":205,"spinlock_wait_cycles":5248,"thread_start_us":82,"threads_started":1}
I20260926 16:42:45.783181 29584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.777003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33344 (local address 127.21.12.66:37159)
0926 16:42:45.777379 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:45.777392 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:45.777416 (+ 24us) server_negotiation.cc:407] Connection header received
0926 16:42:45.777482 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:45.777486 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:45.777652 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:45.777977 (+ 325us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:45.778132 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.779244 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.780983 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.781799 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.782753 (+ 954us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:45.782910 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:45.782956 (+ 46us) server_negotiation.cc:299] Negotiation successful
0926 16:42:45.783036 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":121,"threads_started":1}
I20260926 16:42:45.785022 29206 tablet_service.cc:1467] Processing CreateTablet for tablet ec1ef19bbdaa46c88f97692bef7d984c (DEFAULT_TABLE table=test-table [id=455cbfda11bf4195923ce1a1a710753a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:45.785356 29351 tablet_service.cc:1467] Processing CreateTablet for tablet ec1ef19bbdaa46c88f97692bef7d984c (DEFAULT_TABLE table=test-table [id=455cbfda11bf4195923ce1a1a710753a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:45.785681 29491 tablet_service.cc:1467] Processing CreateTablet for tablet ec1ef19bbdaa46c88f97692bef7d984c (DEFAULT_TABLE table=test-table [id=455cbfda11bf4195923ce1a1a710753a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:45.786157 29206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec1ef19bbdaa46c88f97692bef7d984c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:45.786180 29351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec1ef19bbdaa46c88f97692bef7d984c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:45.786572 29491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec1ef19bbdaa46c88f97692bef7d984c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:45.792893 29585 tablet_bootstrap.cc:492] T ec1ef19bbdaa46c88f97692bef7d984c P 1abc5ef5f6924b23a687a4aa4df47af6: Bootstrap starting.
I20260926 16:42:45.793869 29586 tablet_bootstrap.cc:492] T ec1ef19bbdaa46c88f97692bef7d984c P a870f8287ca3464b8746958bd1fe700f: Bootstrap starting.
I20260926 16:42:45.793869 29587 tablet_bootstrap.cc:492] T ec1ef19bbdaa46c88f97692bef7d984c P d35b2f782ab34e35b9e401ed869c808c: Bootstrap starting.
I20260926 16:42:45.795104 29585 tablet_bootstrap.cc:654] T ec1ef19bbdaa46c88f97692bef7d984c P 1abc5ef5f6924b23a687a4aa4df47af6: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:45.795825 29585 log.cc:824] T ec1ef19bbdaa46c88f97692bef7d984c P 1abc5ef5f6924b23a687a4aa4df47af6: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:45.795980 29586 tablet_bootstrap.cc:654] T ec1ef19bbdaa46c88f97692bef7d984c P a870f8287ca3464b8746958bd1fe700f: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:45.795980 29587 tablet_bootstrap.cc:654] T ec1ef19bbdaa46c88f97692bef7d984c P d35b2f782ab34e35b9e401ed869c808c: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:45.796818 29587 log.cc:824] T ec1ef19bbdaa46c88f97692bef7d984c P d35b2f782ab34e35b9e401ed869c808c: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:45.796818 29586 log.cc:824] T ec1ef19bbdaa46c88f97692bef7d984c P a870f8287ca3464b8746958bd1fe700f: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:45.797873 29585 tablet_bootstrap.cc:492] T ec1ef19bbdaa46c88f97692bef7d984c P 1abc5ef5f6924b23a687a4aa4df47af6: No bootstrap required, opened a new log
I20260926 16:42:45.798076 29585 ts_tablet_manager.cc:1397] T ec1ef19bbdaa46c88f97692bef7d984c P 1abc5ef5f6924b23a687a4aa4df47af6: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260926 16:42:45.798645 29586 tablet_bootstrap.cc:492] T ec1ef19bbdaa46c88f97692bef7d984c P a870f8287ca3464b8746958bd1fe700f: No bootstrap required, opened a new log
I20260926 16:42:45.798820 29586 ts_tablet_manager.cc:1397] T ec1ef19bbdaa46c88f97692bef7d984c P a870f8287ca3464b8746958bd1fe700f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 16:42:45.801009 29585 raft_consensus.cc:348] T ec1ef19bbdaa46c88f97692bef7d984c P 1abc5ef5f6924b23a687a4aa4df47af6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } }
I20260926 16:42:45.801247 29585 raft_consensus.cc:374] T ec1ef19bbdaa46c88f97692bef7d984c P 1abc5ef5f6924b23a687a4aa4df47af6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:45.801303 29585 raft_consensus.cc:729] T ec1ef19bbdaa46c88f97692bef7d984c P 1abc5ef5f6924b23a687a4aa4df47af6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1abc5ef5f6924b23a687a4aa4df47af6, State: Initialized, Role: FOLLOWER
I20260926 16:42:45.801743 29585 consensus_queue.cc:260] T ec1ef19bbdaa46c88f97692bef7d984c P 1abc5ef5f6924b23a687a4aa4df47af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } }
I20260926 16:42:45.802446 29417 heartbeater.cc:503] Master 127.21.12.126:33749 was elected leader, sending a full tablet report...
I20260926 16:42:45.802714 29587 tablet_bootstrap.cc:492] T ec1ef19bbdaa46c88f97692bef7d984c P d35b2f782ab34e35b9e401ed869c808c: No bootstrap required, opened a new log
I20260926 16:42:45.802783 29585 ts_tablet_manager.cc:1428] T ec1ef19bbdaa46c88f97692bef7d984c P 1abc5ef5f6924b23a687a4aa4df47af6: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260926 16:42:45.802893 29587 ts_tablet_manager.cc:1397] T ec1ef19bbdaa46c88f97692bef7d984c P d35b2f782ab34e35b9e401ed869c808c: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260926 16:42:45.802917 29586 raft_consensus.cc:348] T ec1ef19bbdaa46c88f97692bef7d984c P a870f8287ca3464b8746958bd1fe700f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } }
I20260926 16:42:45.803167 29586 raft_consensus.cc:374] T ec1ef19bbdaa46c88f97692bef7d984c P a870f8287ca3464b8746958bd1fe700f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:45.803225 29586 raft_consensus.cc:729] T ec1ef19bbdaa46c88f97692bef7d984c P a870f8287ca3464b8746958bd1fe700f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a870f8287ca3464b8746958bd1fe700f, State: Initialized, Role: FOLLOWER
I20260926 16:42:45.803700 29586 consensus_queue.cc:260] T ec1ef19bbdaa46c88f97692bef7d984c P a870f8287ca3464b8746958bd1fe700f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } }
I20260926 16:42:45.804332 29277 heartbeater.cc:503] Master 127.21.12.126:33749 was elected leader, sending a full tablet report...
I20260926 16:42:45.804667 29586 ts_tablet_manager.cc:1428] T ec1ef19bbdaa46c88f97692bef7d984c P a870f8287ca3464b8746958bd1fe700f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 16:42:45.807288 29587 raft_consensus.cc:348] T ec1ef19bbdaa46c88f97692bef7d984c P d35b2f782ab34e35b9e401ed869c808c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } }
I20260926 16:42:45.807560 29587 raft_consensus.cc:374] T ec1ef19bbdaa46c88f97692bef7d984c P d35b2f782ab34e35b9e401ed869c808c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:45.807617 29587 raft_consensus.cc:729] T ec1ef19bbdaa46c88f97692bef7d984c P d35b2f782ab34e35b9e401ed869c808c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35b2f782ab34e35b9e401ed869c808c, State: Initialized, Role: FOLLOWER
I20260926 16:42:45.808004 29587 consensus_queue.cc:260] T ec1ef19bbdaa46c88f97692bef7d984c P d35b2f782ab34e35b9e401ed869c808c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } }
I20260926 16:42:45.808581 29557 heartbeater.cc:503] Master 127.21.12.126:33749 was elected leader, sending a full tablet report...
I20260926 16:42:45.808876 29587 ts_tablet_manager.cc:1428] T ec1ef19bbdaa46c88f97692bef7d984c P d35b2f782ab34e35b9e401ed869c808c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 16:42:45.924261 29592 raft_consensus.cc:482] T ec1ef19bbdaa46c88f97692bef7d984c P a870f8287ca3464b8746958bd1fe700f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:42:45.924491 29592 raft_consensus.cc:504] T ec1ef19bbdaa46c88f97692bef7d984c P a870f8287ca3464b8746958bd1fe700f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } }
I20260926 16:42:45.925694 29592 leader_election.cc:288] T ec1ef19bbdaa46c88f97692bef7d984c P a870f8287ca3464b8746958bd1fe700f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d35b2f782ab34e35b9e401ed869c808c (127.21.12.67:42921), 1abc5ef5f6924b23a687a4aa4df47af6 (127.21.12.66:37159)
I20260926 16:42:45.930109 29584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.926593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:56207 (local address 127.21.12.66:37159)
0926 16:42:45.926716 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:45.926719 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:45.926909 (+ 190us) server_negotiation.cc:407] Connection header received
0926 16:42:45.927030 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:45.927036 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:45.927088 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:45.927147 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:45.927424 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.928016 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.929056 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.929499 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.929590 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:45.929676 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:45.929902 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 16:42:45.929984 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 16:42:45.930101 29595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.926395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:37159 (local address 127.21.12.65:56207)
0926 16:42:45.926817 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:45.926829 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:45.926931 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:45.927180 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:45.927182 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:45.927200 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:45.927318 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:45.927324 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.928122 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.928125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:45.928915 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:45.928924 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.929614 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.929619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:45.929719 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:45.929796 (+ 77us) client_negotiation.cc:769] Sending connection context
0926 16:42:45.929866 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 16:42:45.929920 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":20,"thread_start_us":85,"threads_started":1}
I20260926 16:42:45.930248 29582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.926035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:36497 (local address 127.21.12.67:42921)
0926 16:42:45.926185 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:45.926190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:45.926442 (+ 252us) server_negotiation.cc:407] Connection header received
0926 16:42:45.926503 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:45.926506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:45.926559 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:45.926640 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:45.927087 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.927743 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.929019 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.929568 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.929685 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:45.929742 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:45.930028 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 16:42:45.930129 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 16:42:45.930294 29594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.925923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.67:42921 (local address 127.21.12.65:36497)
0926 16:42:45.926253 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:45.926270 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:45.926385 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:45.926757 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:45.926760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:45.926778 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:45.926910 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:45.926916 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.927857 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.927860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:45.928889 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:45.928899 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.929703 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.929706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:45.929894 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:45.929919 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 16:42:45.929995 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 16:42:45.930061 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":5,"thread_start_us":171,"threads_started":1}
I20260926 16:42:45.930980 29511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec1ef19bbdaa46c88f97692bef7d984c" candidate_uuid: "a870f8287ca3464b8746958bd1fe700f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35b2f782ab34e35b9e401ed869c808c" is_pre_election: true
I20260926 16:42:45.930984 29371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec1ef19bbdaa46c88f97692bef7d984c" candidate_uuid: "a870f8287ca3464b8746958bd1fe700f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" is_pre_election: true
I20260926 16:42:45.931284 29511 raft_consensus.cc:2454] T ec1ef19bbdaa46c88f97692bef7d984c P d35b2f782ab34e35b9e401ed869c808c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a870f8287ca3464b8746958bd1fe700f in term 0.
I20260926 16:42:45.931298 29371 raft_consensus.cc:2454] T ec1ef19bbdaa46c88f97692bef7d984c P 1abc5ef5f6924b23a687a4aa4df47af6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a870f8287ca3464b8746958bd1fe700f in term 0.
I20260926 16:42:45.931778 29163 leader_election.cc:302] T ec1ef19bbdaa46c88f97692bef7d984c P a870f8287ca3464b8746958bd1fe700f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a870f8287ca3464b8746958bd1fe700f, d35b2f782ab34e35b9e401ed869c808c; no voters:
I20260926 16:42:45.932063 29592 raft_consensus.cc:2781] T ec1ef19bbdaa46c88f97692bef7d984c P a870f8287ca3464b8746958bd1fe700f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:42:45.932154 29592 raft_consensus.cc:482] T ec1ef19bbdaa46c88f97692bef7d984c P a870f8287ca3464b8746958bd1fe700f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:42:45.932199 29592 raft_consensus.cc:3037] T ec1ef19bbdaa46c88f97692bef7d984c P a870f8287ca3464b8746958bd1fe700f [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:45.933193 29592 raft_consensus.cc:504] T ec1ef19bbdaa46c88f97692bef7d984c P a870f8287ca3464b8746958bd1fe700f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } }
I20260926 16:42:45.933521 29592 leader_election.cc:288] T ec1ef19bbdaa46c88f97692bef7d984c P a870f8287ca3464b8746958bd1fe700f [CANDIDATE]: Term 1 election: Requested vote from peers d35b2f782ab34e35b9e401ed869c808c (127.21.12.67:42921), 1abc5ef5f6924b23a687a4aa4df47af6 (127.21.12.66:37159)
I20260926 16:42:45.933856 29511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec1ef19bbdaa46c88f97692bef7d984c" candidate_uuid: "a870f8287ca3464b8746958bd1fe700f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35b2f782ab34e35b9e401ed869c808c"
I20260926 16:42:45.933990 29511 raft_consensus.cc:3037] T ec1ef19bbdaa46c88f97692bef7d984c P d35b2f782ab34e35b9e401ed869c808c [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:45.934003 29371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec1ef19bbdaa46c88f97692bef7d984c" candidate_uuid: "a870f8287ca3464b8746958bd1fe700f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1abc5ef5f6924b23a687a4aa4df47af6"
I20260926 16:42:45.934142 29371 raft_consensus.cc:3037] T ec1ef19bbdaa46c88f97692bef7d984c P 1abc5ef5f6924b23a687a4aa4df47af6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:45.935045 29511 raft_consensus.cc:2454] T ec1ef19bbdaa46c88f97692bef7d984c P d35b2f782ab34e35b9e401ed869c808c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a870f8287ca3464b8746958bd1fe700f in term 1.
I20260926 16:42:45.935319 29163 leader_election.cc:302] T ec1ef19bbdaa46c88f97692bef7d984c P a870f8287ca3464b8746958bd1fe700f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a870f8287ca3464b8746958bd1fe700f, d35b2f782ab34e35b9e401ed869c808c; no voters:
I20260926 16:42:45.935493 29592 raft_consensus.cc:2781] T ec1ef19bbdaa46c88f97692bef7d984c P a870f8287ca3464b8746958bd1fe700f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:45.935724 29371 raft_consensus.cc:2454] T ec1ef19bbdaa46c88f97692bef7d984c P 1abc5ef5f6924b23a687a4aa4df47af6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a870f8287ca3464b8746958bd1fe700f in term 1.
I20260926 16:42:45.935741 29592 raft_consensus.cc:686] T ec1ef19bbdaa46c88f97692bef7d984c P a870f8287ca3464b8746958bd1fe700f [term 1 LEADER]: Becoming Leader. State: Replica: a870f8287ca3464b8746958bd1fe700f, State: Running, Role: LEADER
I20260926 16:42:45.936092 29592 consensus_queue.cc:237] T ec1ef19bbdaa46c88f97692bef7d984c P a870f8287ca3464b8746958bd1fe700f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } }
I20260926 16:42:45.939190 29074 catalog_manager.cc:5495] T ec1ef19bbdaa46c88f97692bef7d984c P a870f8287ca3464b8746958bd1fe700f reported cstate change: term changed from 0 to 1, leader changed from <none> to a870f8287ca3464b8746958bd1fe700f (127.21.12.65). New cstate: current_term: 1 leader_uuid: "a870f8287ca3464b8746958bd1fe700f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } health_report { overall_health: UNKNOWN } } }
W20260926 16:42:45.961668 29558 tablet.cc:2369] T ec1ef19bbdaa46c88f97692bef7d984c P d35b2f782ab34e35b9e401ed869c808c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:42:45.971262 29581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.967785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58740 (local address 127.21.12.65:38091)
0926 16:42:45.967940 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:45.967944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:45.967957 (+ 13us) server_negotiation.cc:407] Connection header received
0926 16:42:45.968035 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:45.968037 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:45.968091 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:45.968189 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:42:45.968407 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.968934 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.969784 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.970007 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.970118 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:45.970391 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:42:45.970835 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:42:45.970925 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:45.971028 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 16:42:45.971080 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 16:42:45.977016 29511 raft_consensus.cc:1261] T ec1ef19bbdaa46c88f97692bef7d984c P d35b2f782ab34e35b9e401ed869c808c [term 1 FOLLOWER]: Refusing update from remote peer a870f8287ca3464b8746958bd1fe700f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:42:45.977032 29371 raft_consensus.cc:1261] T ec1ef19bbdaa46c88f97692bef7d984c P 1abc5ef5f6924b23a687a4aa4df47af6 [term 1 FOLLOWER]: Refusing update from remote peer a870f8287ca3464b8746958bd1fe700f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:42:45.977793 29592 consensus_queue.cc:1035] T ec1ef19bbdaa46c88f97692bef7d984c P a870f8287ca3464b8746958bd1fe700f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:42:45.977983 29596 consensus_queue.cc:1035] T ec1ef19bbdaa46c88f97692bef7d984c P a870f8287ca3464b8746958bd1fe700f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 16:42:45.984994 29418 tablet.cc:2369] T ec1ef19bbdaa46c88f97692bef7d984c P 1abc5ef5f6924b23a687a4aa4df47af6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:42:45.987445 29601 mvcc.cc:205] Tried to move back new op lower bound from 7333646196634820608 to 7333646196477149184. Current Snapshot: MvccSnapshot[applied={T|T < 7333646196634820608}]
I20260926 16:42:45.995146 29074 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:42:45.995306 29074 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:42:45.997663 29074 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a2017701436402f9ebb08a9813ae014: Sending DeleteTablet for 3 replicas of tablet ec1ef19bbdaa46c88f97692bef7d984c
I20260926 16:42:45.998467 29206 tablet_service.cc:1514] Processing DeleteTablet for tablet ec1ef19bbdaa46c88f97692bef7d984c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:45 UTC) from {username='oryx', principal='oryx/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:58730
I20260926 16:42:45.998560 29491 tablet_service.cc:1514] Processing DeleteTablet for tablet ec1ef19bbdaa46c88f97692bef7d984c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:45 UTC) from {username='oryx', principal='oryx/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:33050
I20260926 16:42:45.998667 29351 tablet_service.cc:1514] Processing DeleteTablet for tablet ec1ef19bbdaa46c88f97692bef7d984c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:45 UTC) from {username='oryx', principal='oryx/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:33344
I20260926 16:42:45.999054 29610 tablet_replica.cc:330] T ec1ef19bbdaa46c88f97692bef7d984c P 1abc5ef5f6924b23a687a4aa4df47af6: stopping tablet replica
I20260926 16:42:45.999348 29610 raft_consensus.cc:2229] T ec1ef19bbdaa46c88f97692bef7d984c P 1abc5ef5f6924b23a687a4aa4df47af6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:42:45.999783 29610 raft_consensus.cc:2258] T ec1ef19bbdaa46c88f97692bef7d984c P 1abc5ef5f6924b23a687a4aa4df47af6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:42:45.999846 29074 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47216:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:42:46.000214 29074 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:42:46.001225 29608 tablet_replica.cc:330] T ec1ef19bbdaa46c88f97692bef7d984c P a870f8287ca3464b8746958bd1fe700f: stopping tablet replica
I20260926 16:42:46.001236 29610 ts_tablet_manager.cc:1905] T ec1ef19bbdaa46c88f97692bef7d984c P 1abc5ef5f6924b23a687a4aa4df47af6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:42:46.001509 29608 raft_consensus.cc:2229] T ec1ef19bbdaa46c88f97692bef7d984c P a870f8287ca3464b8746958bd1fe700f [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:42:46.001907 29608 raft_consensus.cc:2258] T ec1ef19bbdaa46c88f97692bef7d984c P a870f8287ca3464b8746958bd1fe700f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:42:46.002240 29609 tablet_replica.cc:330] T ec1ef19bbdaa46c88f97692bef7d984c P d35b2f782ab34e35b9e401ed869c808c: stopping tablet replica
I20260926 16:42:46.002508 29609 raft_consensus.cc:2229] T ec1ef19bbdaa46c88f97692bef7d984c P d35b2f782ab34e35b9e401ed869c808c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:42:46.002811 29609 raft_consensus.cc:2258] T ec1ef19bbdaa46c88f97692bef7d984c P d35b2f782ab34e35b9e401ed869c808c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:42:46.003196 29608 ts_tablet_manager.cc:1905] T ec1ef19bbdaa46c88f97692bef7d984c P a870f8287ca3464b8746958bd1fe700f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:42:46.003980 29610 ts_tablet_manager.cc:1918] T ec1ef19bbdaa46c88f97692bef7d984c P 1abc5ef5f6924b23a687a4aa4df47af6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:42:46.004086 29610 log.cc:1196] T ec1ef19bbdaa46c88f97692bef7d984c P 1abc5ef5f6924b23a687a4aa4df47af6: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-1/wal/wals/ec1ef19bbdaa46c88f97692bef7d984c
I20260926 16:42:46.004376 29610 ts_tablet_manager.cc:1939] T ec1ef19bbdaa46c88f97692bef7d984c P 1abc5ef5f6924b23a687a4aa4df47af6: Deleting consensus metadata
I20260926 16:42:46.004379 29609 ts_tablet_manager.cc:1905] T ec1ef19bbdaa46c88f97692bef7d984c P d35b2f782ab34e35b9e401ed869c808c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:42:46.005892 29608 ts_tablet_manager.cc:1918] T ec1ef19bbdaa46c88f97692bef7d984c P a870f8287ca3464b8746958bd1fe700f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:42:46.005986 29608 log.cc:1196] T ec1ef19bbdaa46c88f97692bef7d984c P a870f8287ca3464b8746958bd1fe700f: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-0/wal/wals/ec1ef19bbdaa46c88f97692bef7d984c
I20260926 16:42:46.006287 29608 ts_tablet_manager.cc:1939] T ec1ef19bbdaa46c88f97692bef7d984c P a870f8287ca3464b8746958bd1fe700f: Deleting consensus metadata
I20260926 16:42:46.006350 29206 tablet_service.cc:1467] Processing CreateTablet for tablet 3697a15ea0f3423fbc0ea56ed1b9ea48 (DEFAULT_TABLE table=test-table [id=8f87f36e1aa04198855444bb74a5df78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:46.006470 29059 catalog_manager.cc:4841] TS 1abc5ef5f6924b23a687a4aa4df47af6 (127.21.12.66:37159): tablet ec1ef19bbdaa46c88f97692bef7d984c (table test-table [id=455cbfda11bf4195923ce1a1a710753a]) successfully deleted
I20260926 16:42:46.006682 29206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3697a15ea0f3423fbc0ea56ed1b9ea48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:46.006940 29491 tablet_service.cc:1467] Processing CreateTablet for tablet 3697a15ea0f3423fbc0ea56ed1b9ea48 (DEFAULT_TABLE table=test-table [id=8f87f36e1aa04198855444bb74a5df78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:46.007174 29609 ts_tablet_manager.cc:1918] T ec1ef19bbdaa46c88f97692bef7d984c P d35b2f782ab34e35b9e401ed869c808c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:42:46.007262 29609 log.cc:1196] T ec1ef19bbdaa46c88f97692bef7d984c P d35b2f782ab34e35b9e401ed869c808c: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-2/wal/wals/ec1ef19bbdaa46c88f97692bef7d984c
I20260926 16:42:46.007553 29061 catalog_manager.cc:4841] TS a870f8287ca3464b8746958bd1fe700f (127.21.12.65:38091): tablet ec1ef19bbdaa46c88f97692bef7d984c (table test-table [id=455cbfda11bf4195923ce1a1a710753a]) successfully deleted
I20260926 16:42:46.007627 29609 ts_tablet_manager.cc:1939] T ec1ef19bbdaa46c88f97692bef7d984c P d35b2f782ab34e35b9e401ed869c808c: Deleting consensus metadata
I20260926 16:42:46.007874 29351 tablet_service.cc:1467] Processing CreateTablet for tablet 3697a15ea0f3423fbc0ea56ed1b9ea48 (DEFAULT_TABLE table=test-table [id=8f87f36e1aa04198855444bb74a5df78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:46.008095 29351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3697a15ea0f3423fbc0ea56ed1b9ea48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:46.007195 29491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3697a15ea0f3423fbc0ea56ed1b9ea48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:46.008785 29586 tablet_bootstrap.cc:492] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P a870f8287ca3464b8746958bd1fe700f: Bootstrap starting.
I20260926 16:42:46.009861 29586 tablet_bootstrap.cc:654] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P a870f8287ca3464b8746958bd1fe700f: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:46.010396 29585 tablet_bootstrap.cc:492] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P 1abc5ef5f6924b23a687a4aa4df47af6: Bootstrap starting.
I20260926 16:42:46.010888 29587 tablet_bootstrap.cc:492] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P d35b2f782ab34e35b9e401ed869c808c: Bootstrap starting.
I20260926 16:42:46.011271 29585 tablet_bootstrap.cc:654] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P 1abc5ef5f6924b23a687a4aa4df47af6: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:46.011768 29587 tablet_bootstrap.cc:654] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P d35b2f782ab34e35b9e401ed869c808c: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:46.012336 29059 catalog_manager.cc:4841] TS d35b2f782ab34e35b9e401ed869c808c (127.21.12.67:42921): tablet ec1ef19bbdaa46c88f97692bef7d984c (table test-table [id=455cbfda11bf4195923ce1a1a710753a]) successfully deleted
I20260926 16:42:46.012836 29585 tablet_bootstrap.cc:492] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P 1abc5ef5f6924b23a687a4aa4df47af6: No bootstrap required, opened a new log
I20260926 16:42:46.012940 29585 ts_tablet_manager.cc:1397] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P 1abc5ef5f6924b23a687a4aa4df47af6: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260926 16:42:46.013309 29585 raft_consensus.cc:348] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P 1abc5ef5f6924b23a687a4aa4df47af6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } }
I20260926 16:42:46.013427 29585 raft_consensus.cc:374] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P 1abc5ef5f6924b23a687a4aa4df47af6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:46.013470 29585 raft_consensus.cc:729] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P 1abc5ef5f6924b23a687a4aa4df47af6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1abc5ef5f6924b23a687a4aa4df47af6, State: Initialized, Role: FOLLOWER
I20260926 16:42:46.013597 29585 consensus_queue.cc:260] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P 1abc5ef5f6924b23a687a4aa4df47af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } }
I20260926 16:42:46.013914 29585 ts_tablet_manager.cc:1428] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P 1abc5ef5f6924b23a687a4aa4df47af6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 16:42:46.015291 29587 tablet_bootstrap.cc:492] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P d35b2f782ab34e35b9e401ed869c808c: No bootstrap required, opened a new log
I20260926 16:42:46.015369 29587 ts_tablet_manager.cc:1397] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P d35b2f782ab34e35b9e401ed869c808c: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260926 16:42:46.015363 29586 tablet_bootstrap.cc:492] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P a870f8287ca3464b8746958bd1fe700f: No bootstrap required, opened a new log
I20260926 16:42:46.015439 29586 ts_tablet_manager.cc:1397] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P a870f8287ca3464b8746958bd1fe700f: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20260926 16:42:46.015767 29587 raft_consensus.cc:348] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P d35b2f782ab34e35b9e401ed869c808c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } }
I20260926 16:42:46.015868 29587 raft_consensus.cc:374] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P d35b2f782ab34e35b9e401ed869c808c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:46.015898 29587 raft_consensus.cc:729] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P d35b2f782ab34e35b9e401ed869c808c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35b2f782ab34e35b9e401ed869c808c, State: Initialized, Role: FOLLOWER
I20260926 16:42:46.015905 29586 raft_consensus.cc:348] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P a870f8287ca3464b8746958bd1fe700f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } }
I20260926 16:42:46.016026 29586 raft_consensus.cc:374] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P a870f8287ca3464b8746958bd1fe700f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:46.016001 29587 consensus_queue.cc:260] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P d35b2f782ab34e35b9e401ed869c808c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } }
I20260926 16:42:46.016060 29586 raft_consensus.cc:729] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P a870f8287ca3464b8746958bd1fe700f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a870f8287ca3464b8746958bd1fe700f, State: Initialized, Role: FOLLOWER
I20260926 16:42:46.016227 29587 ts_tablet_manager.cc:1428] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P d35b2f782ab34e35b9e401ed869c808c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 16:42:46.016184 29586 consensus_queue.cc:260] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P a870f8287ca3464b8746958bd1fe700f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } }
I20260926 16:42:46.016464 29586 ts_tablet_manager.cc:1428] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P a870f8287ca3464b8746958bd1fe700f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20260926 16:42:46.260008 29278 tablet.cc:2369] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P a870f8287ca3464b8746958bd1fe700f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:42:46.273516 29592 raft_consensus.cc:482] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P a870f8287ca3464b8746958bd1fe700f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:42:46.273641 29592 raft_consensus.cc:504] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P a870f8287ca3464b8746958bd1fe700f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } }
I20260926 16:42:46.274067 29592 leader_election.cc:288] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P a870f8287ca3464b8746958bd1fe700f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1abc5ef5f6924b23a687a4aa4df47af6 (127.21.12.66:37159), d35b2f782ab34e35b9e401ed869c808c (127.21.12.67:42921)
I20260926 16:42:46.274573 29371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3697a15ea0f3423fbc0ea56ed1b9ea48" candidate_uuid: "a870f8287ca3464b8746958bd1fe700f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" is_pre_election: true
I20260926 16:42:46.274739 29371 raft_consensus.cc:2454] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P 1abc5ef5f6924b23a687a4aa4df47af6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a870f8287ca3464b8746958bd1fe700f in term 0.
I20260926 16:42:46.274705 29511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3697a15ea0f3423fbc0ea56ed1b9ea48" candidate_uuid: "a870f8287ca3464b8746958bd1fe700f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35b2f782ab34e35b9e401ed869c808c" is_pre_election: true
I20260926 16:42:46.274894 29511 raft_consensus.cc:2454] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P d35b2f782ab34e35b9e401ed869c808c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a870f8287ca3464b8746958bd1fe700f in term 0.
I20260926 16:42:46.275063 29163 leader_election.cc:302] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P a870f8287ca3464b8746958bd1fe700f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1abc5ef5f6924b23a687a4aa4df47af6, a870f8287ca3464b8746958bd1fe700f; no voters:
I20260926 16:42:46.275247 29592 raft_consensus.cc:2781] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P a870f8287ca3464b8746958bd1fe700f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:42:46.275317 29592 raft_consensus.cc:482] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P a870f8287ca3464b8746958bd1fe700f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:42:46.275346 29592 raft_consensus.cc:3037] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P a870f8287ca3464b8746958bd1fe700f [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:46.276309 29592 raft_consensus.cc:504] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P a870f8287ca3464b8746958bd1fe700f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } }
I20260926 16:42:46.276634 29592 leader_election.cc:288] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P a870f8287ca3464b8746958bd1fe700f [CANDIDATE]: Term 1 election: Requested vote from peers 1abc5ef5f6924b23a687a4aa4df47af6 (127.21.12.66:37159), d35b2f782ab34e35b9e401ed869c808c (127.21.12.67:42921)
I20260926 16:42:46.277024 29371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3697a15ea0f3423fbc0ea56ed1b9ea48" candidate_uuid: "a870f8287ca3464b8746958bd1fe700f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1abc5ef5f6924b23a687a4aa4df47af6"
I20260926 16:42:46.277151 29371 raft_consensus.cc:3037] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P 1abc5ef5f6924b23a687a4aa4df47af6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:46.277118 29511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3697a15ea0f3423fbc0ea56ed1b9ea48" candidate_uuid: "a870f8287ca3464b8746958bd1fe700f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35b2f782ab34e35b9e401ed869c808c"
I20260926 16:42:46.277251 29511 raft_consensus.cc:3037] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P d35b2f782ab34e35b9e401ed869c808c [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:46.278190 29371 raft_consensus.cc:2454] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P 1abc5ef5f6924b23a687a4aa4df47af6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a870f8287ca3464b8746958bd1fe700f in term 1.
I20260926 16:42:46.278216 29511 raft_consensus.cc:2454] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P d35b2f782ab34e35b9e401ed869c808c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a870f8287ca3464b8746958bd1fe700f in term 1.
I20260926 16:42:46.278478 29163 leader_election.cc:302] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P a870f8287ca3464b8746958bd1fe700f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a870f8287ca3464b8746958bd1fe700f, d35b2f782ab34e35b9e401ed869c808c; no voters:
I20260926 16:42:46.278685 29592 raft_consensus.cc:2781] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P a870f8287ca3464b8746958bd1fe700f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:46.278798 29592 raft_consensus.cc:686] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P a870f8287ca3464b8746958bd1fe700f [term 1 LEADER]: Becoming Leader. State: Replica: a870f8287ca3464b8746958bd1fe700f, State: Running, Role: LEADER
I20260926 16:42:46.278934 29592 consensus_queue.cc:237] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P a870f8287ca3464b8746958bd1fe700f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } }
I20260926 16:42:46.280543 29070 catalog_manager.cc:5495] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P a870f8287ca3464b8746958bd1fe700f reported cstate change: term changed from 0 to 1, leader changed from <none> to a870f8287ca3464b8746958bd1fe700f (127.21.12.65). New cstate: current_term: 1 leader_uuid: "a870f8287ca3464b8746958bd1fe700f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } health_report { overall_health: UNKNOWN } } }
Sep 26 16:42:46 dist-test-slave-n1hw krb5kdc[29029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440964, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.126@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM
I20260926 16:42:46.336207 29621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:46.327417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:33749 (local address 127.0.0.1:47230)
0926 16:42:46.327804 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:46.327823 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:46.327969 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:46.328235 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:46.328238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:46.328762 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:46.328849 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:46.328856 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.329673 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.329676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:46.330245 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:46.330253 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.330682 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.330686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:46.330757 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:46.331320 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:46.331332 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:46.332751 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:46.334511 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:46.334519 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:46.334524 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:46.334753 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:46.335119 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:46.335121 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:46.335123 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:46.335214 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:46.335699 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:46.335705 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:46.335805 (+ 100us) client_negotiation.cc:769] Sending connection context
0926 16:42:46.335900 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 16:42:46.336028 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":120,"threads_started":1}
I20260926 16:42:46.336228 29620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:46.327446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47230 (local address 127.21.12.126:33749)
0926 16:42:46.327800 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:46.327808 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:46.327906 (+ 98us) server_negotiation.cc:407] Connection header received
0926 16:42:46.328083 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:46.328086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:46.328123 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:46.328216 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:46.328963 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.329526 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.330360 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.330577 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.330682 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:46.332889 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:46.332892 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:46.332894 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:46.332917 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:46.334398 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:46.334843 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:46.334846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:46.334847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:46.334925 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:46.335331 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:46.335334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:46.335335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:46.335586 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:46.335697 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:46.335960 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 16:42:46.336088 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":299,"thread_start_us":107,"threads_started":1}
I20260926 16:42:46.340009 29070 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:47230:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 16:42:46.340662 29070 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:42:46.345573 29206 tablet_service.cc:1467] Processing CreateTablet for tablet 504b5d545bda4978b076d46b0f1c109d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d76851719930498e89730967878e7482]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 16:42:46.345575 29351 tablet_service.cc:1467] Processing CreateTablet for tablet 504b5d545bda4978b076d46b0f1c109d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d76851719930498e89730967878e7482]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 16:42:46.345633 29491 tablet_service.cc:1467] Processing CreateTablet for tablet 504b5d545bda4978b076d46b0f1c109d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d76851719930498e89730967878e7482]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 16:42:46.346009 29206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 504b5d545bda4978b076d46b0f1c109d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:46.346009 29491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 504b5d545bda4978b076d46b0f1c109d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:46.346009 29351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 504b5d545bda4978b076d46b0f1c109d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:46.348239 29586 tablet_bootstrap.cc:492] T 504b5d545bda4978b076d46b0f1c109d P a870f8287ca3464b8746958bd1fe700f: Bootstrap starting.
I20260926 16:42:46.348848 29587 tablet_bootstrap.cc:492] T 504b5d545bda4978b076d46b0f1c109d P d35b2f782ab34e35b9e401ed869c808c: Bootstrap starting.
I20260926 16:42:46.349045 29586 tablet_bootstrap.cc:654] T 504b5d545bda4978b076d46b0f1c109d P a870f8287ca3464b8746958bd1fe700f: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:46.349865 29587 tablet_bootstrap.cc:654] T 504b5d545bda4978b076d46b0f1c109d P d35b2f782ab34e35b9e401ed869c808c: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:46.349893 29585 tablet_bootstrap.cc:492] T 504b5d545bda4978b076d46b0f1c109d P 1abc5ef5f6924b23a687a4aa4df47af6: Bootstrap starting.
I20260926 16:42:46.350675 29585 tablet_bootstrap.cc:654] T 504b5d545bda4978b076d46b0f1c109d P 1abc5ef5f6924b23a687a4aa4df47af6: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:46.350986 29587 tablet_bootstrap.cc:492] T 504b5d545bda4978b076d46b0f1c109d P d35b2f782ab34e35b9e401ed869c808c: No bootstrap required, opened a new log
I20260926 16:42:46.351089 29587 ts_tablet_manager.cc:1397] T 504b5d545bda4978b076d46b0f1c109d P d35b2f782ab34e35b9e401ed869c808c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260926 16:42:46.351594 29587 raft_consensus.cc:348] T 504b5d545bda4978b076d46b0f1c109d P d35b2f782ab34e35b9e401ed869c808c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } }
I20260926 16:42:46.351713 29587 raft_consensus.cc:374] T 504b5d545bda4978b076d46b0f1c109d P d35b2f782ab34e35b9e401ed869c808c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:46.351758 29587 raft_consensus.cc:729] T 504b5d545bda4978b076d46b0f1c109d P d35b2f782ab34e35b9e401ed869c808c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35b2f782ab34e35b9e401ed869c808c, State: Initialized, Role: FOLLOWER
I20260926 16:42:46.351858 29585 tablet_bootstrap.cc:492] T 504b5d545bda4978b076d46b0f1c109d P 1abc5ef5f6924b23a687a4aa4df47af6: No bootstrap required, opened a new log
I20260926 16:42:46.351902 29586 tablet_bootstrap.cc:492] T 504b5d545bda4978b076d46b0f1c109d P a870f8287ca3464b8746958bd1fe700f: No bootstrap required, opened a new log
I20260926 16:42:46.351938 29585 ts_tablet_manager.cc:1397] T 504b5d545bda4978b076d46b0f1c109d P 1abc5ef5f6924b23a687a4aa4df47af6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 16:42:46.351891 29587 consensus_queue.cc:260] T 504b5d545bda4978b076d46b0f1c109d P d35b2f782ab34e35b9e401ed869c808c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } }
I20260926 16:42:46.351995 29586 ts_tablet_manager.cc:1397] T 504b5d545bda4978b076d46b0f1c109d P a870f8287ca3464b8746958bd1fe700f: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260926 16:42:46.352217 29593 tablet_replica.cc:439] T 504b5d545bda4978b076d46b0f1c109d P d35b2f782ab34e35b9e401ed869c808c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } } }
I20260926 16:42:46.352376 29593 tablet_replica.cc:442] T 504b5d545bda4978b076d46b0f1c109d P d35b2f782ab34e35b9e401ed869c808c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:42:46.352406 29585 raft_consensus.cc:348] T 504b5d545bda4978b076d46b0f1c109d P 1abc5ef5f6924b23a687a4aa4df47af6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } }
I20260926 16:42:46.352440 29586 raft_consensus.cc:348] T 504b5d545bda4978b076d46b0f1c109d P a870f8287ca3464b8746958bd1fe700f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } }
I20260926 16:42:46.352520 29585 raft_consensus.cc:374] T 504b5d545bda4978b076d46b0f1c109d P 1abc5ef5f6924b23a687a4aa4df47af6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:46.352557 29585 raft_consensus.cc:729] T 504b5d545bda4978b076d46b0f1c109d P 1abc5ef5f6924b23a687a4aa4df47af6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1abc5ef5f6924b23a687a4aa4df47af6, State: Initialized, Role: FOLLOWER
I20260926 16:42:46.352563 29586 raft_consensus.cc:374] T 504b5d545bda4978b076d46b0f1c109d P a870f8287ca3464b8746958bd1fe700f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:46.352593 29586 raft_consensus.cc:729] T 504b5d545bda4978b076d46b0f1c109d P a870f8287ca3464b8746958bd1fe700f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a870f8287ca3464b8746958bd1fe700f, State: Initialized, Role: FOLLOWER
I20260926 16:42:46.352680 29585 consensus_queue.cc:260] T 504b5d545bda4978b076d46b0f1c109d P 1abc5ef5f6924b23a687a4aa4df47af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } }
I20260926 16:42:46.352728 29586 consensus_queue.cc:260] T 504b5d545bda4978b076d46b0f1c109d P a870f8287ca3464b8746958bd1fe700f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } }
I20260926 16:42:46.352941 29585 ts_tablet_manager.cc:1428] T 504b5d545bda4978b076d46b0f1c109d P 1abc5ef5f6924b23a687a4aa4df47af6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 16:42:46.352978 29586 ts_tablet_manager.cc:1428] T 504b5d545bda4978b076d46b0f1c109d P a870f8287ca3464b8746958bd1fe700f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 16:42:46.353443 29591 tablet_replica.cc:439] T 504b5d545bda4978b076d46b0f1c109d P 1abc5ef5f6924b23a687a4aa4df47af6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } } }
I20260926 16:42:46.353601 29591 tablet_replica.cc:442] T 504b5d545bda4978b076d46b0f1c109d P 1abc5ef5f6924b23a687a4aa4df47af6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:42:46.353678 29592 tablet_replica.cc:439] T 504b5d545bda4978b076d46b0f1c109d P a870f8287ca3464b8746958bd1fe700f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } } }
I20260926 16:42:46.353847 29592 tablet_replica.cc:442] T 504b5d545bda4978b076d46b0f1c109d P a870f8287ca3464b8746958bd1fe700f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:42:46.354724 29587 ts_tablet_manager.cc:1428] T 504b5d545bda4978b076d46b0f1c109d P d35b2f782ab34e35b9e401ed869c808c: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260926 16:42:46.518507 29591 raft_consensus.cc:482] T 504b5d545bda4978b076d46b0f1c109d P 1abc5ef5f6924b23a687a4aa4df47af6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:42:46.518638 29591 raft_consensus.cc:504] T 504b5d545bda4978b076d46b0f1c109d P 1abc5ef5f6924b23a687a4aa4df47af6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } }
I20260926 16:42:46.519742 29591 leader_election.cc:288] T 504b5d545bda4978b076d46b0f1c109d P 1abc5ef5f6924b23a687a4aa4df47af6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a870f8287ca3464b8746958bd1fe700f (127.21.12.65:38091), d35b2f782ab34e35b9e401ed869c808c (127.21.12.67:42921)
I20260926 16:42:46.525159 29627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:46.520149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:45381 (local address 127.21.12.67:42921)
0926 16:42:46.520404 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:46.520410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:46.520639 (+ 229us) server_negotiation.cc:407] Connection header received
0926 16:42:46.520723 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:46.520729 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:46.520779 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:46.520864 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:46.521116 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.521968 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.523786 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.524408 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.524563 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:46.524608 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:46.524939 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 16:42:46.525019 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":65,"threads_started":1}
I20260926 16:42:46.525159 29626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:46.520147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:33837 (local address 127.21.12.65:38091)
0926 16:42:46.520398 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:46.520403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:46.520420 (+ 17us) server_negotiation.cc:407] Connection header received
0926 16:42:46.520570 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:46.520573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:46.520643 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:46.520742 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:46.521095 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.521918 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.523776 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.524402 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.524545 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:46.524598 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:46.524945 (+ 347us) server_negotiation.cc:299] Negotiation successful
0926 16:42:46.525019 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":72,"threads_started":1}
I20260926 16:42:46.525163 29628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:46.520038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.67:42921 (local address 127.21.12.66:45381)
0926 16:42:46.520550 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:46.520562 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:46.520645 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:46.520887 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:46.520892 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:46.520905 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:46.521008 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:46.521017 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.522061 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.522063 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:46.523630 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:46.523641 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.524557 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.524561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:46.524760 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:46.524821 (+ 61us) client_negotiation.cc:769] Sending connection context
0926 16:42:46.524899 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 16:42:46.524988 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"mutex_wait_us":471,"thread_start_us":65,"threads_started":1}
I20260926 16:42:46.525163 29625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:46.520040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:38091 (local address 127.21.12.66:33837)
0926 16:42:46.520322 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:46.520338 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:46.520453 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:46.520764 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:46.520766 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:46.520778 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:46.520957 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:46.520964 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.522054 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.522057 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:46.523630 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:46.523641 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.524557 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.524561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:46.524758 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:46.524821 (+ 63us) client_negotiation.cc:769] Sending connection context
0926 16:42:46.524900 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 16:42:46.524988 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":207,"thread_start_us":79,"threads_started":1}
I20260926 16:42:46.525816 29511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "504b5d545bda4978b076d46b0f1c109d" candidate_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35b2f782ab34e35b9e401ed869c808c" is_pre_election: true
I20260926 16:42:46.526017 29511 raft_consensus.cc:2454] T 504b5d545bda4978b076d46b0f1c109d P d35b2f782ab34e35b9e401ed869c808c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1abc5ef5f6924b23a687a4aa4df47af6 in term 0.
I20260926 16:42:46.526130 29231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "504b5d545bda4978b076d46b0f1c109d" candidate_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a870f8287ca3464b8746958bd1fe700f" is_pre_election: true
I20260926 16:42:46.526372 29231 raft_consensus.cc:2454] T 504b5d545bda4978b076d46b0f1c109d P a870f8287ca3464b8746958bd1fe700f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1abc5ef5f6924b23a687a4aa4df47af6 in term 0.
I20260926 16:42:46.526393 29303 leader_election.cc:302] T 504b5d545bda4978b076d46b0f1c109d P 1abc5ef5f6924b23a687a4aa4df47af6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1abc5ef5f6924b23a687a4aa4df47af6, d35b2f782ab34e35b9e401ed869c808c; no voters:
I20260926 16:42:46.526693 29591 raft_consensus.cc:2781] T 504b5d545bda4978b076d46b0f1c109d P 1abc5ef5f6924b23a687a4aa4df47af6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:42:46.526767 29591 raft_consensus.cc:482] T 504b5d545bda4978b076d46b0f1c109d P 1abc5ef5f6924b23a687a4aa4df47af6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:42:46.526805 29591 raft_consensus.cc:3037] T 504b5d545bda4978b076d46b0f1c109d P 1abc5ef5f6924b23a687a4aa4df47af6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:46.527768 29591 raft_consensus.cc:504] T 504b5d545bda4978b076d46b0f1c109d P 1abc5ef5f6924b23a687a4aa4df47af6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } }
I20260926 16:42:46.528107 29591 leader_election.cc:288] T 504b5d545bda4978b076d46b0f1c109d P 1abc5ef5f6924b23a687a4aa4df47af6 [CANDIDATE]: Term 1 election: Requested vote from peers a870f8287ca3464b8746958bd1fe700f (127.21.12.65:38091), d35b2f782ab34e35b9e401ed869c808c (127.21.12.67:42921)
I20260926 16:42:46.528407 29231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "504b5d545bda4978b076d46b0f1c109d" candidate_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a870f8287ca3464b8746958bd1fe700f"
I20260926 16:42:46.528549 29231 raft_consensus.cc:3037] T 504b5d545bda4978b076d46b0f1c109d P a870f8287ca3464b8746958bd1fe700f [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:46.528518 29511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "504b5d545bda4978b076d46b0f1c109d" candidate_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35b2f782ab34e35b9e401ed869c808c"
I20260926 16:42:46.528646 29511 raft_consensus.cc:3037] T 504b5d545bda4978b076d46b0f1c109d P d35b2f782ab34e35b9e401ed869c808c [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:46.529654 29511 raft_consensus.cc:2454] T 504b5d545bda4978b076d46b0f1c109d P d35b2f782ab34e35b9e401ed869c808c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1abc5ef5f6924b23a687a4aa4df47af6 in term 1.
I20260926 16:42:46.529686 29231 raft_consensus.cc:2454] T 504b5d545bda4978b076d46b0f1c109d P a870f8287ca3464b8746958bd1fe700f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1abc5ef5f6924b23a687a4aa4df47af6 in term 1.
I20260926 16:42:46.529983 29305 leader_election.cc:302] T 504b5d545bda4978b076d46b0f1c109d P 1abc5ef5f6924b23a687a4aa4df47af6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1abc5ef5f6924b23a687a4aa4df47af6, a870f8287ca3464b8746958bd1fe700f; no voters:
I20260926 16:42:46.530164 29591 raft_consensus.cc:2781] T 504b5d545bda4978b076d46b0f1c109d P 1abc5ef5f6924b23a687a4aa4df47af6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:46.530357 29591 raft_consensus.cc:686] T 504b5d545bda4978b076d46b0f1c109d P 1abc5ef5f6924b23a687a4aa4df47af6 [term 1 LEADER]: Becoming Leader. State: Replica: 1abc5ef5f6924b23a687a4aa4df47af6, State: Running, Role: LEADER
I20260926 16:42:46.530611 29591 consensus_queue.cc:237] T 504b5d545bda4978b076d46b0f1c109d P 1abc5ef5f6924b23a687a4aa4df47af6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } }
I20260926 16:42:46.531805 29629 tablet_replica.cc:439] T 504b5d545bda4978b076d46b0f1c109d P 1abc5ef5f6924b23a687a4aa4df47af6: TxnStatusTablet state changed. Reason: New leader 1abc5ef5f6924b23a687a4aa4df47af6. Latest consensus state: current_term: 1 leader_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } } }
I20260926 16:42:46.531919 29629 tablet_replica.cc:442] T 504b5d545bda4978b076d46b0f1c109d P 1abc5ef5f6924b23a687a4aa4df47af6: This TxnStatusTablet replica's current role is: LEADER
I20260926 16:42:46.532171 29631 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 16:42:46.532320 29631 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 16:42:46.533113 29073 catalog_manager.cc:5495] T 504b5d545bda4978b076d46b0f1c109d P 1abc5ef5f6924b23a687a4aa4df47af6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1abc5ef5f6924b23a687a4aa4df47af6 (127.21.12.66). New cstate: current_term: 1 leader_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } health_report { overall_health: UNKNOWN } } }
I20260926 16:42:46.546795 29621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:46.542702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:37159 (local address 127.0.0.1:33356)
0926 16:42:46.542850 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:46.542867 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:46.543008 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:46.543381 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:46.543385 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:46.543393 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 16:42:46.543507 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:46.543529 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.544386 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.544392 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:46.545067 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:46.545076 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.545592 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.545596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:46.545696 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:46.545795 (+ 99us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 16:42:46.546477 (+ 682us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 16:42:46.546494 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 16:42:46.546581 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 16:42:46.546672 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 16:42:46.547214 29634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:46.542784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33356 (local address 127.21.12.66:37159)
0926 16:42:46.543065 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:46.543078 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:46.543095 (+ 17us) server_negotiation.cc:407] Connection header received
0926 16:42:46.543163 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:46.543166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:46.543240 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:46.543368 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:42:46.543641 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.544262 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.545194 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.545470 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.545595 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:46.545964 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:42:46.546366 (+ 402us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:42:46.546499 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:46.546591 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 16:42:46.546644 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":96,"threads_started":1}
I20260926 16:42:46.549876 29511 raft_consensus.cc:1261] T 504b5d545bda4978b076d46b0f1c109d P d35b2f782ab34e35b9e401ed869c808c [term 1 FOLLOWER]: Refusing update from remote peer 1abc5ef5f6924b23a687a4aa4df47af6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:42:46.550179 29231 raft_consensus.cc:1261] T 504b5d545bda4978b076d46b0f1c109d P a870f8287ca3464b8746958bd1fe700f [term 1 FOLLOWER]: Refusing update from remote peer 1abc5ef5f6924b23a687a4aa4df47af6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:42:46.550366 29629 consensus_queue.cc:1035] T 504b5d545bda4978b076d46b0f1c109d P 1abc5ef5f6924b23a687a4aa4df47af6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:42:46.550668 29591 consensus_queue.cc:1035] T 504b5d545bda4978b076d46b0f1c109d P 1abc5ef5f6924b23a687a4aa4df47af6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:42:46.551788 29593 tablet_replica.cc:439] T 504b5d545bda4978b076d46b0f1c109d P d35b2f782ab34e35b9e401ed869c808c: TxnStatusTablet state changed. Reason: New leader 1abc5ef5f6924b23a687a4aa4df47af6. Latest consensus state: current_term: 1 leader_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } } }
I20260926 16:42:46.551904 29593 tablet_replica.cc:442] T 504b5d545bda4978b076d46b0f1c109d P d35b2f782ab34e35b9e401ed869c808c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:42:46.552461 29592 tablet_replica.cc:439] T 504b5d545bda4978b076d46b0f1c109d P a870f8287ca3464b8746958bd1fe700f: TxnStatusTablet state changed. Reason: New leader 1abc5ef5f6924b23a687a4aa4df47af6. Latest consensus state: current_term: 1 leader_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } } }
I20260926 16:42:46.552578 29592 tablet_replica.cc:442] T 504b5d545bda4978b076d46b0f1c109d P a870f8287ca3464b8746958bd1fe700f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:42:46.555434 29629 tablet_replica.cc:439] T 504b5d545bda4978b076d46b0f1c109d P 1abc5ef5f6924b23a687a4aa4df47af6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } } }
I20260926 16:42:46.555546 29629 tablet_replica.cc:442] T 504b5d545bda4978b076d46b0f1c109d P 1abc5ef5f6924b23a687a4aa4df47af6: This TxnStatusTablet replica's current role is: LEADER
I20260926 16:42:46.555656 29593 tablet_replica.cc:439] T 504b5d545bda4978b076d46b0f1c109d P d35b2f782ab34e35b9e401ed869c808c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } } }
I20260926 16:42:46.555759 29593 tablet_replica.cc:442] T 504b5d545bda4978b076d46b0f1c109d P d35b2f782ab34e35b9e401ed869c808c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:42:46.555773 29629 tablet_replica.cc:439] T 504b5d545bda4978b076d46b0f1c109d P 1abc5ef5f6924b23a687a4aa4df47af6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } } }
I20260926 16:42:46.555876 29629 tablet_replica.cc:442] T 504b5d545bda4978b076d46b0f1c109d P 1abc5ef5f6924b23a687a4aa4df47af6: This TxnStatusTablet replica's current role is: LEADER
I20260926 16:42:46.557562 29638 mvcc.cc:205] Tried to move back new op lower bound from 7333646198983389184 to 7333646198911619072. Current Snapshot: MvccSnapshot[applied={T|T < 7333646198983389184}]
I20260926 16:42:46.557734 29592 tablet_replica.cc:439] T 504b5d545bda4978b076d46b0f1c109d P a870f8287ca3464b8746958bd1fe700f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } } }
I20260926 16:42:46.557874 29592 tablet_replica.cc:442] T 504b5d545bda4978b076d46b0f1c109d P a870f8287ca3464b8746958bd1fe700f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:42:46.573652 29634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:46.569848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:36393 (local address 127.21.12.66:37159)
0926 16:42:46.569991 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:46.569998 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:46.570164 (+ 166us) server_negotiation.cc:407] Connection header received
0926 16:42:46.570285 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:46.570288 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:46.570331 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:46.570396 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:42:46.570710 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.571355 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.572635 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.572856 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.572972 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:46.573256 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:42:46.573325 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:42:46.573387 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:46.573498 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 16:42:46.573544 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 16:42:46.573743 29646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:46.569733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:37159 (local address 127.21.12.65:36393)
0926 16:42:46.570073 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:46.570089 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:46.570198 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:46.570428 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:46.570431 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:46.570439 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 16:42:46.570599 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:46.570606 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.571460 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.571464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:46.572483 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:46.572493 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.572973 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.572977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:46.573067 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:46.573120 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 16:42:46.573410 (+ 290us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 16:42:46.573416 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 16:42:46.573476 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 16:42:46.573562 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":115,"threads_started":1}
I20260926 16:42:46.580271 29371 raft_consensus.cc:1261] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P 1abc5ef5f6924b23a687a4aa4df47af6 [term 1 FOLLOWER]: Refusing update from remote peer a870f8287ca3464b8746958bd1fe700f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:42:46.580710 29596 consensus_queue.cc:1035] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P a870f8287ca3464b8746958bd1fe700f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:42:46.580830 29511 raft_consensus.cc:1261] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P d35b2f782ab34e35b9e401ed869c808c [term 1 FOLLOWER]: Refusing update from remote peer a870f8287ca3464b8746958bd1fe700f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:42:46.581310 29596 consensus_queue.cc:1035] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P a870f8287ca3464b8746958bd1fe700f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:42:46.604504 29626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:46.600225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:42581 (local address 127.21.12.65:38091)
0926 16:42:46.600345 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:46.600349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:46.600539 (+ 190us) server_negotiation.cc:407] Connection header received
0926 16:42:46.600774 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:46.600778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:46.600849 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:46.600929 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:42:46.601200 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.602039 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.603226 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.603569 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.603688 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:46.604004 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:42:46.604084 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:42:46.604151 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:46.604292 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 16:42:46.604351 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 16:42:46.604481 29650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:46.600161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:38091 (local address 127.21.12.66:42581)
0926 16:42:46.600458 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:46.600470 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:46.600668 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:46.600941 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:46.600944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:46.600952 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 16:42:46.601081 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:46.601088 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.602156 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.602160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:46.603114 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:46.603124 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.603735 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.603739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:46.603813 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:46.603859 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 16:42:46.604188 (+ 329us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 16:42:46.604194 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 16:42:46.604262 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 16:42:46.604326 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":107,"threads_started":1}
I20260926 16:42:46.630151 29073 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:42:46.630321 29073 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:42:46.632422 29073 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a2017701436402f9ebb08a9813ae014: Sending DeleteTablet for 3 replicas of tablet 3697a15ea0f3423fbc0ea56ed1b9ea48
I20260926 16:42:46.632995 29351 tablet_service.cc:1514] Processing DeleteTablet for tablet 3697a15ea0f3423fbc0ea56ed1b9ea48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:46 UTC) from {username='oryx', principal='oryx/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:33344
I20260926 16:42:46.633026 29206 tablet_service.cc:1514] Processing DeleteTablet for tablet 3697a15ea0f3423fbc0ea56ed1b9ea48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:46 UTC) from {username='oryx', principal='oryx/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:58730
I20260926 16:42:46.633162 29491 tablet_service.cc:1514] Processing DeleteTablet for tablet 3697a15ea0f3423fbc0ea56ed1b9ea48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:46 UTC) from {username='oryx', principal='oryx/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:33050
I20260926 16:42:46.633389 29653 tablet_replica.cc:330] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P a870f8287ca3464b8746958bd1fe700f: stopping tablet replica
I20260926 16:42:46.633510 29655 tablet_replica.cc:330] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P d35b2f782ab34e35b9e401ed869c808c: stopping tablet replica
I20260926 16:42:46.633526 29653 raft_consensus.cc:2229] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P a870f8287ca3464b8746958bd1fe700f [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:42:46.633630 29655 raft_consensus.cc:2229] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P d35b2f782ab34e35b9e401ed869c808c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:42:46.633772 29655 raft_consensus.cc:2258] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P d35b2f782ab34e35b9e401ed869c808c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:42:46.633782 29653 raft_consensus.cc:2258] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P a870f8287ca3464b8746958bd1fe700f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:42:46.634465 29653 ts_tablet_manager.cc:1905] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P a870f8287ca3464b8746958bd1fe700f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:42:46.636511 29655 ts_tablet_manager.cc:1905] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P d35b2f782ab34e35b9e401ed869c808c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:42:46.637616 29654 tablet_replica.cc:330] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P 1abc5ef5f6924b23a687a4aa4df47af6: stopping tablet replica
I20260926 16:42:46.637857 29654 raft_consensus.cc:2229] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P 1abc5ef5f6924b23a687a4aa4df47af6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:42:46.637935 29653 ts_tablet_manager.cc:1918] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P a870f8287ca3464b8746958bd1fe700f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:42:46.638012 29653 log.cc:1196] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P a870f8287ca3464b8746958bd1fe700f: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-0/wal/wals/3697a15ea0f3423fbc0ea56ed1b9ea48
I20260926 16:42:46.638161 29654 raft_consensus.cc:2258] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P 1abc5ef5f6924b23a687a4aa4df47af6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:42:46.638302 29653 ts_tablet_manager.cc:1939] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P a870f8287ca3464b8746958bd1fe700f: Deleting consensus metadata
I20260926 16:42:46.639016 29654 ts_tablet_manager.cc:1905] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P 1abc5ef5f6924b23a687a4aa4df47af6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:42:46.639396 29655 ts_tablet_manager.cc:1918] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P d35b2f782ab34e35b9e401ed869c808c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:42:46.639478 29655 log.cc:1196] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P d35b2f782ab34e35b9e401ed869c808c: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-2/wal/wals/3697a15ea0f3423fbc0ea56ed1b9ea48
I20260926 16:42:46.639756 29655 ts_tablet_manager.cc:1939] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P d35b2f782ab34e35b9e401ed869c808c: Deleting consensus metadata
I20260926 16:42:46.640533 29061 catalog_manager.cc:4841] TS a870f8287ca3464b8746958bd1fe700f (127.21.12.65:38091): tablet 3697a15ea0f3423fbc0ea56ed1b9ea48 (table test-table [id=8f87f36e1aa04198855444bb74a5df78]) successfully deleted
I20260926 16:42:46.640738 29059 catalog_manager.cc:4841] TS d35b2f782ab34e35b9e401ed869c808c (127.21.12.67:42921): tablet 3697a15ea0f3423fbc0ea56ed1b9ea48 (table test-table [id=8f87f36e1aa04198855444bb74a5df78]) successfully deleted
I20260926 16:42:46.641664 29654 ts_tablet_manager.cc:1918] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P 1abc5ef5f6924b23a687a4aa4df47af6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:42:46.641762 29654 log.cc:1196] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P 1abc5ef5f6924b23a687a4aa4df47af6: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-1/wal/wals/3697a15ea0f3423fbc0ea56ed1b9ea48
I20260926 16:42:46.642036 29654 ts_tablet_manager.cc:1939] T 3697a15ea0f3423fbc0ea56ed1b9ea48 P 1abc5ef5f6924b23a687a4aa4df47af6: Deleting consensus metadata
I20260926 16:42:46.642681 29059 catalog_manager.cc:4841] TS 1abc5ef5f6924b23a687a4aa4df47af6 (127.21.12.66:37159): tablet 3697a15ea0f3423fbc0ea56ed1b9ea48 (table test-table [id=8f87f36e1aa04198855444bb74a5df78]) successfully deleted
Sep 26 16:42:46 dist-test-slave-n1hw krb5kdc[29029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 16:42:46 dist-test-slave-n1hw krb5kdc[29029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:42:46 dist-test-slave-n1hw krb5kdc[29029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
W20260926 16:42:46.665098 29620 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:46.657360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47242 (local address 127.21.12.126:33749)
0926 16:42:46.657503 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:46.657508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:46.657637 (+ 129us) server_negotiation.cc:407] Connection header received
0926 16:42:46.657908 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:46.657911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:46.657955 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:46.658018 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:46.658904 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.659437 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.660622 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.660910 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.661029 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:46.664938 (+ 3909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47242: BlockingRecv error: recv got EOF from 127.0.0.1:47242 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 26 16:42:46 dist-test-slave-n1hw krb5kdc[29029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM
I20260926 16:42:46.677057 29620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:46.667024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47258 (local address 127.21.12.126:33749)
0926 16:42:46.667134 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:46.667148 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:46.667344 (+ 196us) server_negotiation.cc:407] Connection header received
0926 16:42:46.667547 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:46.667550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:46.667602 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:46.667708 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:46.668396 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.668981 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.670066 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.670351 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.670495 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:46.673284 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:46.673289 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:46.673291 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:46.673318 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:46.675097 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:46.675598 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:46.675605 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:46.675607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:46.675712 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:46.676050 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:46.676054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:46.676056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:46.676264 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:46.676422 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:46.676721 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 16:42:46.676861 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 16:42:46.680169 29073 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47258:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:42:46.680668 29073 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:42:46.685456 29206 tablet_service.cc:1467] Processing CreateTablet for tablet 76d33c49a9b14daeb37f2bdc0f6d4ce0 (DEFAULT_TABLE table=test-table [id=05a9d3436a284dd7a30ca19963f6fdb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:46.685564 29351 tablet_service.cc:1467] Processing CreateTablet for tablet 76d33c49a9b14daeb37f2bdc0f6d4ce0 (DEFAULT_TABLE table=test-table [id=05a9d3436a284dd7a30ca19963f6fdb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:46.685803 29206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76d33c49a9b14daeb37f2bdc0f6d4ce0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:46.685830 29351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76d33c49a9b14daeb37f2bdc0f6d4ce0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:46.688162 29586 tablet_bootstrap.cc:492] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P a870f8287ca3464b8746958bd1fe700f: Bootstrap starting.
I20260926 16:42:46.688375 29491 tablet_service.cc:1467] Processing CreateTablet for tablet 76d33c49a9b14daeb37f2bdc0f6d4ce0 (DEFAULT_TABLE table=test-table [id=05a9d3436a284dd7a30ca19963f6fdb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:46.688696 29491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76d33c49a9b14daeb37f2bdc0f6d4ce0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:46.689088 29586 tablet_bootstrap.cc:654] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P a870f8287ca3464b8746958bd1fe700f: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:46.689597 29585 tablet_bootstrap.cc:492] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P 1abc5ef5f6924b23a687a4aa4df47af6: Bootstrap starting.
I20260926 16:42:46.690721 29587 tablet_bootstrap.cc:492] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P d35b2f782ab34e35b9e401ed869c808c: Bootstrap starting.
I20260926 16:42:46.690939 29585 tablet_bootstrap.cc:654] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P 1abc5ef5f6924b23a687a4aa4df47af6: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:46.691636 29587 tablet_bootstrap.cc:654] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P d35b2f782ab34e35b9e401ed869c808c: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:46.692255 29586 tablet_bootstrap.cc:492] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P a870f8287ca3464b8746958bd1fe700f: No bootstrap required, opened a new log
I20260926 16:42:46.692341 29586 ts_tablet_manager.cc:1397] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P a870f8287ca3464b8746958bd1fe700f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 16:42:46.692803 29587 tablet_bootstrap.cc:492] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P d35b2f782ab34e35b9e401ed869c808c: No bootstrap required, opened a new log
I20260926 16:42:46.692728 29586 raft_consensus.cc:348] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P a870f8287ca3464b8746958bd1fe700f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } }
I20260926 16:42:46.692865 29586 raft_consensus.cc:374] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P a870f8287ca3464b8746958bd1fe700f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:46.692903 29587 ts_tablet_manager.cc:1397] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P d35b2f782ab34e35b9e401ed869c808c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 16:42:46.692911 29586 raft_consensus.cc:729] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P a870f8287ca3464b8746958bd1fe700f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a870f8287ca3464b8746958bd1fe700f, State: Initialized, Role: FOLLOWER
I20260926 16:42:46.693051 29586 consensus_queue.cc:260] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P a870f8287ca3464b8746958bd1fe700f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } }
I20260926 16:42:46.693332 29586 ts_tablet_manager.cc:1428] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P a870f8287ca3464b8746958bd1fe700f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 16:42:46.693341 29587 raft_consensus.cc:348] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P d35b2f782ab34e35b9e401ed869c808c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } }
I20260926 16:42:46.693480 29587 raft_consensus.cc:374] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P d35b2f782ab34e35b9e401ed869c808c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:46.693524 29587 raft_consensus.cc:729] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P d35b2f782ab34e35b9e401ed869c808c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35b2f782ab34e35b9e401ed869c808c, State: Initialized, Role: FOLLOWER
I20260926 16:42:46.693657 29587 consensus_queue.cc:260] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P d35b2f782ab34e35b9e401ed869c808c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } }
I20260926 16:42:46.693907 29585 tablet_bootstrap.cc:492] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P 1abc5ef5f6924b23a687a4aa4df47af6: No bootstrap required, opened a new log
I20260926 16:42:46.693943 29587 ts_tablet_manager.cc:1428] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P d35b2f782ab34e35b9e401ed869c808c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 16:42:46.693990 29585 ts_tablet_manager.cc:1397] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P 1abc5ef5f6924b23a687a4aa4df47af6: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 16:42:46.694334 29585 raft_consensus.cc:348] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P 1abc5ef5f6924b23a687a4aa4df47af6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } }
I20260926 16:42:46.694444 29585 raft_consensus.cc:374] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P 1abc5ef5f6924b23a687a4aa4df47af6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:46.694479 29585 raft_consensus.cc:729] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P 1abc5ef5f6924b23a687a4aa4df47af6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1abc5ef5f6924b23a687a4aa4df47af6, State: Initialized, Role: FOLLOWER
I20260926 16:42:46.694603 29585 consensus_queue.cc:260] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P 1abc5ef5f6924b23a687a4aa4df47af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } }
I20260926 16:42:46.697052 29585 ts_tablet_manager.cc:1428] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P 1abc5ef5f6924b23a687a4aa4df47af6: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260926 16:42:46.808018 29652 raft_consensus.cc:482] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P a870f8287ca3464b8746958bd1fe700f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:42:46.808190 29652 raft_consensus.cc:504] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P a870f8287ca3464b8746958bd1fe700f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } }
I20260926 16:42:46.808733 29652 leader_election.cc:288] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P a870f8287ca3464b8746958bd1fe700f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1abc5ef5f6924b23a687a4aa4df47af6 (127.21.12.66:37159), d35b2f782ab34e35b9e401ed869c808c (127.21.12.67:42921)
I20260926 16:42:46.809242 29371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76d33c49a9b14daeb37f2bdc0f6d4ce0" candidate_uuid: "a870f8287ca3464b8746958bd1fe700f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" is_pre_election: true
I20260926 16:42:46.809397 29371 raft_consensus.cc:2454] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P 1abc5ef5f6924b23a687a4aa4df47af6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a870f8287ca3464b8746958bd1fe700f in term 0.
I20260926 16:42:46.809351 29511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76d33c49a9b14daeb37f2bdc0f6d4ce0" candidate_uuid: "a870f8287ca3464b8746958bd1fe700f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35b2f782ab34e35b9e401ed869c808c" is_pre_election: true
I20260926 16:42:46.809520 29511 raft_consensus.cc:2454] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P d35b2f782ab34e35b9e401ed869c808c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a870f8287ca3464b8746958bd1fe700f in term 0.
I20260926 16:42:46.809779 29163 leader_election.cc:302] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P a870f8287ca3464b8746958bd1fe700f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1abc5ef5f6924b23a687a4aa4df47af6, a870f8287ca3464b8746958bd1fe700f; no voters:
I20260926 16:42:46.810000 29652 raft_consensus.cc:2781] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P a870f8287ca3464b8746958bd1fe700f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:42:46.810074 29652 raft_consensus.cc:482] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P a870f8287ca3464b8746958bd1fe700f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:42:46.810138 29652 raft_consensus.cc:3037] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P a870f8287ca3464b8746958bd1fe700f [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:46.811333 29652 raft_consensus.cc:504] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P a870f8287ca3464b8746958bd1fe700f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } }
I20260926 16:42:46.811721 29652 leader_election.cc:288] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P a870f8287ca3464b8746958bd1fe700f [CANDIDATE]: Term 1 election: Requested vote from peers 1abc5ef5f6924b23a687a4aa4df47af6 (127.21.12.66:37159), d35b2f782ab34e35b9e401ed869c808c (127.21.12.67:42921)
I20260926 16:42:46.812114 29511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76d33c49a9b14daeb37f2bdc0f6d4ce0" candidate_uuid: "a870f8287ca3464b8746958bd1fe700f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35b2f782ab34e35b9e401ed869c808c"
I20260926 16:42:46.812153 29371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76d33c49a9b14daeb37f2bdc0f6d4ce0" candidate_uuid: "a870f8287ca3464b8746958bd1fe700f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1abc5ef5f6924b23a687a4aa4df47af6"
I20260926 16:42:46.812235 29511 raft_consensus.cc:3037] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P d35b2f782ab34e35b9e401ed869c808c [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:46.812299 29371 raft_consensus.cc:3037] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P 1abc5ef5f6924b23a687a4aa4df47af6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:46.813607 29511 raft_consensus.cc:2454] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P d35b2f782ab34e35b9e401ed869c808c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a870f8287ca3464b8746958bd1fe700f in term 1.
I20260926 16:42:46.814041 29371 raft_consensus.cc:2454] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P 1abc5ef5f6924b23a687a4aa4df47af6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a870f8287ca3464b8746958bd1fe700f in term 1.
I20260926 16:42:46.814165 29163 leader_election.cc:302] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P a870f8287ca3464b8746958bd1fe700f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a870f8287ca3464b8746958bd1fe700f, d35b2f782ab34e35b9e401ed869c808c; no voters:
I20260926 16:42:46.814457 29652 raft_consensus.cc:2781] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P a870f8287ca3464b8746958bd1fe700f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:46.814582 29652 raft_consensus.cc:686] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P a870f8287ca3464b8746958bd1fe700f [term 1 LEADER]: Becoming Leader. State: Replica: a870f8287ca3464b8746958bd1fe700f, State: Running, Role: LEADER
I20260926 16:42:46.814739 29652 consensus_queue.cc:237] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P a870f8287ca3464b8746958bd1fe700f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } }
I20260926 16:42:46.817060 29073 catalog_manager.cc:5495] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P a870f8287ca3464b8746958bd1fe700f reported cstate change: term changed from 0 to 1, leader changed from <none> to a870f8287ca3464b8746958bd1fe700f (127.21.12.65). New cstate: current_term: 1 leader_uuid: "a870f8287ca3464b8746958bd1fe700f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } health_report { overall_health: UNKNOWN } } }
I20260926 16:42:46.855803 29626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:46.851702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58752 (local address 127.21.12.65:38091)
0926 16:42:46.851850 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:46.851855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:46.852057 (+ 202us) server_negotiation.cc:407] Connection header received
0926 16:42:46.852300 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:46.852303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:46.852384 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:46.852471 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:42:46.852809 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.853588 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.854806 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.855050 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.855135 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:46.855393 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:42:46.855472 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:42:46.855520 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:46.855592 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 16:42:46.855634 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 16:42:46.857900 29511 raft_consensus.cc:1261] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P d35b2f782ab34e35b9e401ed869c808c [term 1 FOLLOWER]: Refusing update from remote peer a870f8287ca3464b8746958bd1fe700f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:42:46.857985 29371 raft_consensus.cc:1261] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P 1abc5ef5f6924b23a687a4aa4df47af6 [term 1 FOLLOWER]: Refusing update from remote peer a870f8287ca3464b8746958bd1fe700f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:42:46.858384 29596 consensus_queue.cc:1035] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P a870f8287ca3464b8746958bd1fe700f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:42:46.858600 29652 consensus_queue.cc:1035] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P a870f8287ca3464b8746958bd1fe700f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:42:46.865106 29074 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47258:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:42:46.865245 29074 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47258:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:42:46.867164 29074 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a2017701436402f9ebb08a9813ae014: Sending DeleteTablet for 3 replicas of tablet 76d33c49a9b14daeb37f2bdc0f6d4ce0
I20260926 16:42:46.867682 29206 tablet_service.cc:1514] Processing DeleteTablet for tablet 76d33c49a9b14daeb37f2bdc0f6d4ce0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:46 UTC) from {username='oryx', principal='oryx/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:58730
I20260926 16:42:46.867784 29491 tablet_service.cc:1514] Processing DeleteTablet for tablet 76d33c49a9b14daeb37f2bdc0f6d4ce0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:46 UTC) from {username='oryx', principal='oryx/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:33050
I20260926 16:42:46.867785 29351 tablet_service.cc:1514] Processing DeleteTablet for tablet 76d33c49a9b14daeb37f2bdc0f6d4ce0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:46 UTC) from {username='oryx', principal='oryx/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:33344
I20260926 16:42:46.867954 29655 tablet_replica.cc:330] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P d35b2f782ab34e35b9e401ed869c808c: stopping tablet replica
I20260926 16:42:46.867954 29653 tablet_replica.cc:330] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P a870f8287ca3464b8746958bd1fe700f: stopping tablet replica
I20260926 16:42:46.868114 29653 raft_consensus.cc:2229] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P a870f8287ca3464b8746958bd1fe700f [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:42:46.868122 29655 raft_consensus.cc:2229] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P d35b2f782ab34e35b9e401ed869c808c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:42:46.868242 29655 raft_consensus.cc:2258] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P d35b2f782ab34e35b9e401ed869c808c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:42:46.868307 29653 raft_consensus.cc:2258] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P a870f8287ca3464b8746958bd1fe700f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:42:46.868477 29654 tablet_replica.cc:330] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P 1abc5ef5f6924b23a687a4aa4df47af6: stopping tablet replica
I20260926 16:42:46.868589 29654 raft_consensus.cc:2229] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P 1abc5ef5f6924b23a687a4aa4df47af6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:42:46.868728 29654 raft_consensus.cc:2258] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P 1abc5ef5f6924b23a687a4aa4df47af6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:42:46.868870 29655 ts_tablet_manager.cc:1905] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P d35b2f782ab34e35b9e401ed869c808c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:42:46.869313 29653 ts_tablet_manager.cc:1905] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P a870f8287ca3464b8746958bd1fe700f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:42:46.869333 29074 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47258:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 16:42:46.869554 29654 ts_tablet_manager.cc:1905] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P 1abc5ef5f6924b23a687a4aa4df47af6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 16:42:46.869709 29074 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:42:46.871553 29655 ts_tablet_manager.cc:1918] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P d35b2f782ab34e35b9e401ed869c808c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:42:46.871635 29655 log.cc:1196] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P d35b2f782ab34e35b9e401ed869c808c: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-2/wal/wals/76d33c49a9b14daeb37f2bdc0f6d4ce0
I20260926 16:42:46.871810 29653 ts_tablet_manager.cc:1918] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P a870f8287ca3464b8746958bd1fe700f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:42:46.871876 29653 log.cc:1196] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P a870f8287ca3464b8746958bd1fe700f: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-0/wal/wals/76d33c49a9b14daeb37f2bdc0f6d4ce0
I20260926 16:42:46.871903 29655 ts_tablet_manager.cc:1939] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P d35b2f782ab34e35b9e401ed869c808c: Deleting consensus metadata
I20260926 16:42:46.872078 29654 ts_tablet_manager.cc:1918] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P 1abc5ef5f6924b23a687a4aa4df47af6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:42:46.872133 29653 ts_tablet_manager.cc:1939] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P a870f8287ca3464b8746958bd1fe700f: Deleting consensus metadata
I20260926 16:42:46.872175 29654 log.cc:1196] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P 1abc5ef5f6924b23a687a4aa4df47af6: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-1/wal/wals/76d33c49a9b14daeb37f2bdc0f6d4ce0
I20260926 16:42:46.872450 29654 ts_tablet_manager.cc:1939] T 76d33c49a9b14daeb37f2bdc0f6d4ce0 P 1abc5ef5f6924b23a687a4aa4df47af6: Deleting consensus metadata
I20260926 16:42:46.872598 29059 catalog_manager.cc:4841] TS d35b2f782ab34e35b9e401ed869c808c (127.21.12.67:42921): tablet 76d33c49a9b14daeb37f2bdc0f6d4ce0 (table test-table [id=05a9d3436a284dd7a30ca19963f6fdb2]) successfully deleted
I20260926 16:42:46.872822 29061 catalog_manager.cc:4841] TS a870f8287ca3464b8746958bd1fe700f (127.21.12.65:38091): tablet 76d33c49a9b14daeb37f2bdc0f6d4ce0 (table test-table [id=05a9d3436a284dd7a30ca19963f6fdb2]) successfully deleted
I20260926 16:42:46.873016 29059 catalog_manager.cc:4841] TS 1abc5ef5f6924b23a687a4aa4df47af6 (127.21.12.66:37159): tablet 76d33c49a9b14daeb37f2bdc0f6d4ce0 (table test-table [id=05a9d3436a284dd7a30ca19963f6fdb2]) successfully deleted
I20260926 16:42:46.874683 29206 tablet_service.cc:1467] Processing CreateTablet for tablet 004d6c4bf19b43bb9cf5ee055b5b862b (DEFAULT_TABLE table=test-table [id=916086319865466682ad99b7d06db687]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:46.874734 29491 tablet_service.cc:1467] Processing CreateTablet for tablet 004d6c4bf19b43bb9cf5ee055b5b862b (DEFAULT_TABLE table=test-table [id=916086319865466682ad99b7d06db687]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:46.874773 29351 tablet_service.cc:1467] Processing CreateTablet for tablet 004d6c4bf19b43bb9cf5ee055b5b862b (DEFAULT_TABLE table=test-table [id=916086319865466682ad99b7d06db687]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:46.874930 29206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 004d6c4bf19b43bb9cf5ee055b5b862b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:46.874930 29491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 004d6c4bf19b43bb9cf5ee055b5b862b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:46.875015 29351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 004d6c4bf19b43bb9cf5ee055b5b862b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:46.876605 29585 tablet_bootstrap.cc:492] T 004d6c4bf19b43bb9cf5ee055b5b862b P 1abc5ef5f6924b23a687a4aa4df47af6: Bootstrap starting.
I20260926 16:42:46.876973 29586 tablet_bootstrap.cc:492] T 004d6c4bf19b43bb9cf5ee055b5b862b P a870f8287ca3464b8746958bd1fe700f: Bootstrap starting.
I20260926 16:42:46.877332 29585 tablet_bootstrap.cc:654] T 004d6c4bf19b43bb9cf5ee055b5b862b P 1abc5ef5f6924b23a687a4aa4df47af6: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:46.877532 29586 tablet_bootstrap.cc:654] T 004d6c4bf19b43bb9cf5ee055b5b862b P a870f8287ca3464b8746958bd1fe700f: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:46.878088 29587 tablet_bootstrap.cc:492] T 004d6c4bf19b43bb9cf5ee055b5b862b P d35b2f782ab34e35b9e401ed869c808c: Bootstrap starting.
I20260926 16:42:46.878328 29585 tablet_bootstrap.cc:492] T 004d6c4bf19b43bb9cf5ee055b5b862b P 1abc5ef5f6924b23a687a4aa4df47af6: No bootstrap required, opened a new log
I20260926 16:42:46.878424 29585 ts_tablet_manager.cc:1397] T 004d6c4bf19b43bb9cf5ee055b5b862b P 1abc5ef5f6924b23a687a4aa4df47af6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 16:42:46.878746 29585 raft_consensus.cc:348] T 004d6c4bf19b43bb9cf5ee055b5b862b P 1abc5ef5f6924b23a687a4aa4df47af6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } }
I20260926 16:42:46.878856 29585 raft_consensus.cc:374] T 004d6c4bf19b43bb9cf5ee055b5b862b P 1abc5ef5f6924b23a687a4aa4df47af6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:46.878901 29585 raft_consensus.cc:729] T 004d6c4bf19b43bb9cf5ee055b5b862b P 1abc5ef5f6924b23a687a4aa4df47af6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1abc5ef5f6924b23a687a4aa4df47af6, State: Initialized, Role: FOLLOWER
I20260926 16:42:46.878937 29587 tablet_bootstrap.cc:654] T 004d6c4bf19b43bb9cf5ee055b5b862b P d35b2f782ab34e35b9e401ed869c808c: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:46.879029 29585 consensus_queue.cc:260] T 004d6c4bf19b43bb9cf5ee055b5b862b P 1abc5ef5f6924b23a687a4aa4df47af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } }
I20260926 16:42:46.879281 29585 ts_tablet_manager.cc:1428] T 004d6c4bf19b43bb9cf5ee055b5b862b P 1abc5ef5f6924b23a687a4aa4df47af6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 16:42:46.879802 29586 tablet_bootstrap.cc:492] T 004d6c4bf19b43bb9cf5ee055b5b862b P a870f8287ca3464b8746958bd1fe700f: No bootstrap required, opened a new log
I20260926 16:42:46.879889 29586 ts_tablet_manager.cc:1397] T 004d6c4bf19b43bb9cf5ee055b5b862b P a870f8287ca3464b8746958bd1fe700f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 16:42:46.880244 29586 raft_consensus.cc:348] T 004d6c4bf19b43bb9cf5ee055b5b862b P a870f8287ca3464b8746958bd1fe700f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } }
I20260926 16:42:46.880350 29586 raft_consensus.cc:374] T 004d6c4bf19b43bb9cf5ee055b5b862b P a870f8287ca3464b8746958bd1fe700f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:46.880357 29587 tablet_bootstrap.cc:492] T 004d6c4bf19b43bb9cf5ee055b5b862b P d35b2f782ab34e35b9e401ed869c808c: No bootstrap required, opened a new log
I20260926 16:42:46.880383 29586 raft_consensus.cc:729] T 004d6c4bf19b43bb9cf5ee055b5b862b P a870f8287ca3464b8746958bd1fe700f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a870f8287ca3464b8746958bd1fe700f, State: Initialized, Role: FOLLOWER
I20260926 16:42:46.880440 29587 ts_tablet_manager.cc:1397] T 004d6c4bf19b43bb9cf5ee055b5b862b P d35b2f782ab34e35b9e401ed869c808c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 16:42:46.880498 29586 consensus_queue.cc:260] T 004d6c4bf19b43bb9cf5ee055b5b862b P a870f8287ca3464b8746958bd1fe700f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } }
I20260926 16:42:46.880767 29586 ts_tablet_manager.cc:1428] T 004d6c4bf19b43bb9cf5ee055b5b862b P a870f8287ca3464b8746958bd1fe700f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 16:42:46.880894 29587 raft_consensus.cc:348] T 004d6c4bf19b43bb9cf5ee055b5b862b P d35b2f782ab34e35b9e401ed869c808c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } }
I20260926 16:42:46.881006 29587 raft_consensus.cc:374] T 004d6c4bf19b43bb9cf5ee055b5b862b P d35b2f782ab34e35b9e401ed869c808c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:46.881039 29587 raft_consensus.cc:729] T 004d6c4bf19b43bb9cf5ee055b5b862b P d35b2f782ab34e35b9e401ed869c808c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35b2f782ab34e35b9e401ed869c808c, State: Initialized, Role: FOLLOWER
I20260926 16:42:46.881155 29587 consensus_queue.cc:260] T 004d6c4bf19b43bb9cf5ee055b5b862b P d35b2f782ab34e35b9e401ed869c808c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } }
I20260926 16:42:46.881422 29587 ts_tablet_manager.cc:1428] T 004d6c4bf19b43bb9cf5ee055b5b862b P d35b2f782ab34e35b9e401ed869c808c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 16:42:46.976819 29593 raft_consensus.cc:482] T 004d6c4bf19b43bb9cf5ee055b5b862b P d35b2f782ab34e35b9e401ed869c808c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:42:46.976955 29593 raft_consensus.cc:504] T 004d6c4bf19b43bb9cf5ee055b5b862b P d35b2f782ab34e35b9e401ed869c808c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } }
I20260926 16:42:46.978060 29593 leader_election.cc:288] T 004d6c4bf19b43bb9cf5ee055b5b862b P d35b2f782ab34e35b9e401ed869c808c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a870f8287ca3464b8746958bd1fe700f (127.21.12.65:38091), 1abc5ef5f6924b23a687a4aa4df47af6 (127.21.12.66:37159)
I20260926 16:42:46.981923 29634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:46.978500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:51947 (local address 127.21.12.66:37159)
0926 16:42:46.978616 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:46.978621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:46.978707 (+ 86us) server_negotiation.cc:407] Connection header received
0926 16:42:46.978869 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:46.978874 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:46.978922 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:46.979009 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:46.979319 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.979821 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.980911 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.981324 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.981469 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:46.981524 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:46.981707 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 16:42:46.981810 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 16:42:46.981950 29678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:46.978325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:37159 (local address 127.21.12.67:51947)
0926 16:42:46.978593 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:46.978610 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:46.978745 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:46.979025 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:46.979028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:46.979078 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:46.979204 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:46.979210 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.979953 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.979956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:46.980781 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:46.980790 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.981467 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.981471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:46.981567 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:46.981590 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 16:42:46.981690 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 16:42:46.981779 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"spinlock_wait_cycles":13184,"thread_start_us":86,"threads_started":1}
I20260926 16:42:46.982158 29626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:46.978443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:59431 (local address 127.21.12.65:38091)
0926 16:42:46.978630 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:46.978634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:46.979022 (+ 388us) server_negotiation.cc:407] Connection header received
0926 16:42:46.979197 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:46.979201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:46.979246 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:46.979329 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:46.979590 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.980087 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.981165 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.981584 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.981715 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:46.981787 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:46.981964 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 16:42:46.982044 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260926 16:42:46.982213 29679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:46.978325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:38091 (local address 127.21.12.67:59431)
0926 16:42:46.978919 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:46.978931 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:46.979085 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:46.979348 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:46.979351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:46.979366 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:46.979480 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:46.979485 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.980203 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.980208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:46.980993 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:46.981008 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.981705 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.981709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:46.981831 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:46.981859 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 16:42:46.981947 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 16:42:46.982023 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"mutex_wait_us":318,"spinlock_wait_cycles":26752,"thread_start_us":114,"threads_started":1}
I20260926 16:42:46.982404 29371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "004d6c4bf19b43bb9cf5ee055b5b862b" candidate_uuid: "d35b2f782ab34e35b9e401ed869c808c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" is_pre_election: true
I20260926 16:42:46.982574 29371 raft_consensus.cc:2454] T 004d6c4bf19b43bb9cf5ee055b5b862b P 1abc5ef5f6924b23a687a4aa4df47af6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35b2f782ab34e35b9e401ed869c808c in term 0.
I20260926 16:42:46.982666 29231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "004d6c4bf19b43bb9cf5ee055b5b862b" candidate_uuid: "d35b2f782ab34e35b9e401ed869c808c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a870f8287ca3464b8746958bd1fe700f" is_pre_election: true
I20260926 16:42:46.982811 29231 raft_consensus.cc:2454] T 004d6c4bf19b43bb9cf5ee055b5b862b P a870f8287ca3464b8746958bd1fe700f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35b2f782ab34e35b9e401ed869c808c in term 0.
I20260926 16:42:46.982942 29443 leader_election.cc:302] T 004d6c4bf19b43bb9cf5ee055b5b862b P d35b2f782ab34e35b9e401ed869c808c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1abc5ef5f6924b23a687a4aa4df47af6, d35b2f782ab34e35b9e401ed869c808c; no voters:
I20260926 16:42:46.983209 29593 raft_consensus.cc:2781] T 004d6c4bf19b43bb9cf5ee055b5b862b P d35b2f782ab34e35b9e401ed869c808c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:42:46.983268 29593 raft_consensus.cc:482] T 004d6c4bf19b43bb9cf5ee055b5b862b P d35b2f782ab34e35b9e401ed869c808c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:42:46.983314 29593 raft_consensus.cc:3037] T 004d6c4bf19b43bb9cf5ee055b5b862b P d35b2f782ab34e35b9e401ed869c808c [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:46.984313 29593 raft_consensus.cc:504] T 004d6c4bf19b43bb9cf5ee055b5b862b P d35b2f782ab34e35b9e401ed869c808c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } }
I20260926 16:42:46.984689 29593 leader_election.cc:288] T 004d6c4bf19b43bb9cf5ee055b5b862b P d35b2f782ab34e35b9e401ed869c808c [CANDIDATE]: Term 1 election: Requested vote from peers a870f8287ca3464b8746958bd1fe700f (127.21.12.65:38091), 1abc5ef5f6924b23a687a4aa4df47af6 (127.21.12.66:37159)
I20260926 16:42:46.984977 29231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "004d6c4bf19b43bb9cf5ee055b5b862b" candidate_uuid: "d35b2f782ab34e35b9e401ed869c808c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a870f8287ca3464b8746958bd1fe700f"
I20260926 16:42:46.985008 29371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "004d6c4bf19b43bb9cf5ee055b5b862b" candidate_uuid: "d35b2f782ab34e35b9e401ed869c808c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1abc5ef5f6924b23a687a4aa4df47af6"
I20260926 16:42:46.985095 29231 raft_consensus.cc:3037] T 004d6c4bf19b43bb9cf5ee055b5b862b P a870f8287ca3464b8746958bd1fe700f [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:46.985132 29371 raft_consensus.cc:3037] T 004d6c4bf19b43bb9cf5ee055b5b862b P 1abc5ef5f6924b23a687a4aa4df47af6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:46.986155 29371 raft_consensus.cc:2454] T 004d6c4bf19b43bb9cf5ee055b5b862b P 1abc5ef5f6924b23a687a4aa4df47af6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35b2f782ab34e35b9e401ed869c808c in term 1.
I20260926 16:42:46.986222 29231 raft_consensus.cc:2454] T 004d6c4bf19b43bb9cf5ee055b5b862b P a870f8287ca3464b8746958bd1fe700f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35b2f782ab34e35b9e401ed869c808c in term 1.
I20260926 16:42:46.986501 29445 leader_election.cc:302] T 004d6c4bf19b43bb9cf5ee055b5b862b P d35b2f782ab34e35b9e401ed869c808c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a870f8287ca3464b8746958bd1fe700f, d35b2f782ab34e35b9e401ed869c808c; no voters:
I20260926 16:42:46.986697 29593 raft_consensus.cc:2781] T 004d6c4bf19b43bb9cf5ee055b5b862b P d35b2f782ab34e35b9e401ed869c808c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:46.986879 29593 raft_consensus.cc:686] T 004d6c4bf19b43bb9cf5ee055b5b862b P d35b2f782ab34e35b9e401ed869c808c [term 1 LEADER]: Becoming Leader. State: Replica: d35b2f782ab34e35b9e401ed869c808c, State: Running, Role: LEADER
I20260926 16:42:46.987133 29593 consensus_queue.cc:237] T 004d6c4bf19b43bb9cf5ee055b5b862b P d35b2f782ab34e35b9e401ed869c808c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } }
I20260926 16:42:46.989538 29070 catalog_manager.cc:5495] T 004d6c4bf19b43bb9cf5ee055b5b862b P d35b2f782ab34e35b9e401ed869c808c reported cstate change: term changed from 0 to 1, leader changed from <none> to d35b2f782ab34e35b9e401ed869c808c (127.21.12.67). New cstate: current_term: 1 leader_uuid: "d35b2f782ab34e35b9e401ed869c808c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d35b2f782ab34e35b9e401ed869c808c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42921 } health_report { overall_health: HEALTHY } } }
I20260926 16:42:47.012142 29627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:47.008640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33054 (local address 127.21.12.67:42921)
0926 16:42:47.008755 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:47.008759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:47.008780 (+ 21us) server_negotiation.cc:407] Connection header received
0926 16:42:47.008910 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:47.008913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:47.008964 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:47.009078 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:42:47.009330 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:47.009925 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:47.010784 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:47.010985 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:47.011060 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:47.011352 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:42:47.011800 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:42:47.011885 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:47.011979 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 16:42:47.012026 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 16:42:47.022862 29684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:47.018991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:37159 (local address 127.21.12.67:53143)
0926 16:42:47.019281 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:47.019293 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:47.019380 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:47.019623 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:47.019625 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:47.019630 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 16:42:47.019776 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:47.019783 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:47.020878 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:47.020881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:47.021506 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:47.021514 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:47.022102 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:47.022105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:47.022179 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:47.022226 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 16:42:47.022553 (+ 327us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 16:42:47.022558 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 16:42:47.022606 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 16:42:47.022691 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":105,"threads_started":1}
I20260926 16:42:47.022791 29634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:47.019083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:53143 (local address 127.21.12.66:37159)
0926 16:42:47.019231 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:47.019234 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:47.019363 (+ 129us) server_negotiation.cc:407] Connection header received
0926 16:42:47.019471 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:47.019475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:47.019523 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:47.019604 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:42:47.019901 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:47.020758 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:47.021633 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:47.021988 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:47.022083 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:47.022363 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:42:47.022433 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:42:47.022542 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:47.022632 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 16:42:47.022680 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 16:42:47.028318 29371 raft_consensus.cc:1261] T 004d6c4bf19b43bb9cf5ee055b5b862b P 1abc5ef5f6924b23a687a4aa4df47af6 [term 1 FOLLOWER]: Refusing update from remote peer d35b2f782ab34e35b9e401ed869c808c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:42:47.028388 29231 raft_consensus.cc:1261] T 004d6c4bf19b43bb9cf5ee055b5b862b P a870f8287ca3464b8746958bd1fe700f [term 1 FOLLOWER]: Refusing update from remote peer d35b2f782ab34e35b9e401ed869c808c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:42:47.028752 29680 consensus_queue.cc:1035] T 004d6c4bf19b43bb9cf5ee055b5b862b P d35b2f782ab34e35b9e401ed869c808c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1abc5ef5f6924b23a687a4aa4df47af6" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:42:47.028949 29593 consensus_queue.cc:1035] T 004d6c4bf19b43bb9cf5ee055b5b862b P d35b2f782ab34e35b9e401ed869c808c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a870f8287ca3464b8746958bd1fe700f" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 38091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:42:47.046005 29627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:47.042895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:33183 (local address 127.21.12.67:42921)
0926 16:42:47.043102 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:47.043106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:47.043121 (+ 15us) server_negotiation.cc:407] Connection header received
0926 16:42:47.043167 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:47.043169 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:47.043204 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:47.043292 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:42:47.043519 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:47.044033 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:47.044852 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:47.045157 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:47.045248 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:47.045511 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:42:47.045613 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:42:47.045689 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:47.045836 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 16:42:47.045891 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260926 16:42:47.046056 29650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:47.042792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.67:42921 (local address 127.21.12.66:33183)
0926 16:42:47.042970 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:47.042983 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:47.043063 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:47.043301 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:47.043304 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:47.043311 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 16:42:47.043405 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:47.043410 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:47.044138 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:47.044144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:47.044728 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:47.044735 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:47.045258 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:47.045262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:47.045345 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:47.045398 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 16:42:47.045703 (+ 305us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 16:42:47.045709 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 16:42:47.045825 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 16:42:47.045916 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260926 16:42:47.067720 29070 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47258:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:42:47.067888 29070 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47258:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:42:47.069890 29070 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a2017701436402f9ebb08a9813ae014: Sending DeleteTablet for 3 replicas of tablet 004d6c4bf19b43bb9cf5ee055b5b862b
I20260926 16:42:47.070515 29491 tablet_service.cc:1514] Processing DeleteTablet for tablet 004d6c4bf19b43bb9cf5ee055b5b862b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:47 UTC) from {username='oryx', principal='oryx/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:33050
I20260926 16:42:47.070564 29351 tablet_service.cc:1514] Processing DeleteTablet for tablet 004d6c4bf19b43bb9cf5ee055b5b862b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:47 UTC) from {username='oryx', principal='oryx/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:33344
I20260926 16:42:47.070751 29654 tablet_replica.cc:330] T 004d6c4bf19b43bb9cf5ee055b5b862b P 1abc5ef5f6924b23a687a4aa4df47af6: stopping tablet replica
I20260926 16:42:47.070748 29206 tablet_service.cc:1514] Processing DeleteTablet for tablet 004d6c4bf19b43bb9cf5ee055b5b862b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:47 UTC) from {username='oryx', principal='oryx/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:58730
I20260926 16:42:47.070880 29654 raft_consensus.cc:2229] T 004d6c4bf19b43bb9cf5ee055b5b862b P 1abc5ef5f6924b23a687a4aa4df47af6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:42:47.070927 29655 tablet_replica.cc:330] T 004d6c4bf19b43bb9cf5ee055b5b862b P d35b2f782ab34e35b9e401ed869c808c: stopping tablet replica
I20260926 16:42:47.070999 29654 raft_consensus.cc:2258] T 004d6c4bf19b43bb9cf5ee055b5b862b P 1abc5ef5f6924b23a687a4aa4df47af6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:42:47.071069 29655 raft_consensus.cc:2229] T 004d6c4bf19b43bb9cf5ee055b5b862b P d35b2f782ab34e35b9e401ed869c808c [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:42:47.071362 29655 raft_consensus.cc:2258] T 004d6c4bf19b43bb9cf5ee055b5b862b P d35b2f782ab34e35b9e401ed869c808c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:42:47.071563 29653 tablet_replica.cc:330] T 004d6c4bf19b43bb9cf5ee055b5b862b P a870f8287ca3464b8746958bd1fe700f: stopping tablet replica
I20260926 16:42:47.071693 29653 raft_consensus.cc:2229] T 004d6c4bf19b43bb9cf5ee055b5b862b P a870f8287ca3464b8746958bd1fe700f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:42:47.071825 29653 raft_consensus.cc:2258] T 004d6c4bf19b43bb9cf5ee055b5b862b P a870f8287ca3464b8746958bd1fe700f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:42:47.071936 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 29146
I20260926 16:42:47.073446 29654 ts_tablet_manager.cc:1905] T 004d6c4bf19b43bb9cf5ee055b5b862b P 1abc5ef5f6924b23a687a4aa4df47af6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:42:47.076098 29654 ts_tablet_manager.cc:1918] T 004d6c4bf19b43bb9cf5ee055b5b862b P 1abc5ef5f6924b23a687a4aa4df47af6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:42:47.076198 29654 log.cc:1196] T 004d6c4bf19b43bb9cf5ee055b5b862b P 1abc5ef5f6924b23a687a4aa4df47af6: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-1/wal/wals/004d6c4bf19b43bb9cf5ee055b5b862b
I20260926 16:42:47.076432 29654 ts_tablet_manager.cc:1939] T 004d6c4bf19b43bb9cf5ee055b5b862b P 1abc5ef5f6924b23a687a4aa4df47af6: Deleting consensus metadata
I20260926 16:42:47.077093 29059 catalog_manager.cc:4841] TS 1abc5ef5f6924b23a687a4aa4df47af6 (127.21.12.66:37159): tablet 004d6c4bf19b43bb9cf5ee055b5b862b (table test-table [id=916086319865466682ad99b7d06db687]) successfully deleted
I20260926 16:42:47.077348 29655 ts_tablet_manager.cc:1905] T 004d6c4bf19b43bb9cf5ee055b5b862b P d35b2f782ab34e35b9e401ed869c808c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:42:47.079257 29655 ts_tablet_manager.cc:1918] T 004d6c4bf19b43bb9cf5ee055b5b862b P d35b2f782ab34e35b9e401ed869c808c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:42:47.079349 29655 log.cc:1196] T 004d6c4bf19b43bb9cf5ee055b5b862b P d35b2f782ab34e35b9e401ed869c808c: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440924937351-21553-0/minicluster-data/ts-2/wal/wals/004d6c4bf19b43bb9cf5ee055b5b862b
I20260926 16:42:47.079584 29655 ts_tablet_manager.cc:1939] T 004d6c4bf19b43bb9cf5ee055b5b862b P d35b2f782ab34e35b9e401ed869c808c: Deleting consensus metadata
I20260926 16:42:47.080157 29059 catalog_manager.cc:4841] TS d35b2f782ab34e35b9e401ed869c808c (127.21.12.67:42921): tablet 004d6c4bf19b43bb9cf5ee055b5b862b (table test-table [id=916086319865466682ad99b7d06db687]) successfully deleted
W20260926 16:42:47.084808 29061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.12.65:38091 (error 108)
I20260926 16:42:47.085886 29691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:47.085469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:38091 (local address 127.0.0.1:58766)
0926 16:42:47.085686 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:47.085744 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.65:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":63,"threads_started":1}
I20260926 16:42:47.085891 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 29286
W20260926 16:42:47.086809 29061 catalog_manager.cc:4584] TS a870f8287ca3464b8746958bd1fe700f (127.21.12.65:38091): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 004d6c4bf19b43bb9cf5ee055b5b862b: Network error: Client connection negotiation failed: client connection to 127.21.12.65:38091: connect: Connection refused (error 111)
I20260926 16:42:47.092722 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 29426
I20260926 16:42:47.098487 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 29042
2026-09-26T16:42:47Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3452 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440924937351-21553-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:42:47 dist-test-slave-n1hw krb5kdc[29694](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:42:47 dist-test-slave-n1hw krb5kdc[29694](info): set up 2 sockets
Sep 26 16:42:47 dist-test-slave-n1hw krb5kdc[29694](info): commencing operation
krb5kdc: starting...
W20260926 16:42:48.180656 21553 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Sep 26 16:42:48 dist-test-slave-n1hw krb5kdc[29694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:42:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:42:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
I20260926 16:42:48.300352 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:40679
--webserver_interface=127.21.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:46875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:40679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:48.412207 29707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:48.412443 29707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:48.412482 29707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:48.416590 29707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:42:48.416646 29707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:48.416666 29707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:48.416682 29707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:42:48.416697 29707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:42:48.421995 29707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:46875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:40679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:40679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:48.422978 29707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:48.423992 29707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:48.432301 29712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:48.432416 29707 server_base.cc:1034] running on GCE node
W20260926 16:42:48.432269 29715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:48.432264 29713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:48.433064 29707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:48.434180 29707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:48.435384 29707 hybrid_clock.cc:648] HybridClock initialized: now 1790440968435353 us; error 44 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T16:42:48Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1349 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:42:48 dist-test-slave-n1hw krb5kdc[29720](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:42:48 dist-test-slave-n1hw krb5kdc[29720](info): set up 2 sockets
Sep 26 16:42:48 dist-test-slave-n1hw krb5kdc[29720](info): commencing operation
krb5kdc: starting...
W20260926 16:42:49.488595 21553 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Sep 26 16:42:49 dist-test-slave-n1hw krb5kdc[29720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:42:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:42:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
I20260926 16:42:49.606693 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:38995
--webserver_interface=127.21.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:33869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:38995
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:49.714545 29733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:49.714812 29733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:49.714851 29733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:49.718879 29733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:42:49.718940 29733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:49.718961 29733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:49.718976 29733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:42:49.718991 29733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:42:49.724341 29733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:33869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:38995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:38995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:49.725371 29733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:49.726438 29733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:49.735015 29741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:49.734988 29739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:49.735178 29738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:49.735270 29733 server_base.cc:1034] running on GCE node
I20260926 16:42:49.735904 29733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:49.736996 29733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:49.738198 29733 hybrid_clock.cc:648] HybridClock initialized: now 1790440969738181 us; error 35 us; skew 500 ppm
Sep 26 16:42:49 dist-test-slave-n1hw krb5kdc[29720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440969, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:49.741258 29733 init.cc:377] Logged in from keytab as kudu/127.21.12.126@KRBTEST.COM (short username kudu)
I20260926 16:42:49.742290 29733 webserver.cc:460] Webserver started at http://127.21.12.126:40883/ using document root <none> and password file <none>
I20260926 16:42:49.742897 29733 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:49.742996 29733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:49.743244 29733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:49.745384 29733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "48999f8889dd42d1993aac45ee038fbc"
format_stamp: "Formatted at 2026-09-26 16:42:49 on dist-test-slave-n1hw"
server_key: "24fb9cd80f17063f05d1a0222c41f2e3"
server_key_iv: "67e320f97dd9f1e40072655f80f7ca66"
server_key_version: "encryptionkey@0"
I20260926 16:42:49.746018 29733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "48999f8889dd42d1993aac45ee038fbc"
format_stamp: "Formatted at 2026-09-26 16:42:49 on dist-test-slave-n1hw"
server_key: "24fb9cd80f17063f05d1a0222c41f2e3"
server_key_iv: "67e320f97dd9f1e40072655f80f7ca66"
server_key_version: "encryptionkey@0"
I20260926 16:42:49.749853 29733 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 16:42:49.752327 29748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:49.753531 29733 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:42:49.753690 29733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "48999f8889dd42d1993aac45ee038fbc"
format_stamp: "Formatted at 2026-09-26 16:42:49 on dist-test-slave-n1hw"
server_key: "24fb9cd80f17063f05d1a0222c41f2e3"
server_key_iv: "67e320f97dd9f1e40072655f80f7ca66"
server_key_version: "encryptionkey@0"
I20260926 16:42:49.753858 29733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:49.792802 29733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:49.796150 29733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:49.796355 29733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:49.803388 29733 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.126:38995
I20260926 16:42:49.803464 29800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.126:38995 every 8 connection(s)
I20260926 16:42:49.804419 29733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:42:49.807200 29801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:49.812603 29801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48999f8889dd42d1993aac45ee038fbc: Bootstrap starting.
I20260926 16:42:49.812608 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 29733
I20260926 16:42:49.812754 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:42:49.813058 21553 external_mini_cluster.cc:1428] Setting key 0ed1b6f2253d2c152ffb8a08066bd8c9
I20260926 16:42:49.815531 29801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48999f8889dd42d1993aac45ee038fbc: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:49.816252 29801 log.cc:824] T 00000000000000000000000000000000 P 48999f8889dd42d1993aac45ee038fbc: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:49.818285 29801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48999f8889dd42d1993aac45ee038fbc: No bootstrap required, opened a new log
Sep 26 16:42:49 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:42:49.822762 29801 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48999f8889dd42d1993aac45ee038fbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48999f8889dd42d1993aac45ee038fbc" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 38995 } }
I20260926 16:42:49.822986 29801 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48999f8889dd42d1993aac45ee038fbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:49.823064 29801 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48999f8889dd42d1993aac45ee038fbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48999f8889dd42d1993aac45ee038fbc, State: Initialized, Role: FOLLOWER
I20260926 16:42:49.823567 29801 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48999f8889dd42d1993aac45ee038fbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48999f8889dd42d1993aac45ee038fbc" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 38995 } }
I20260926 16:42:49.823762 29801 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48999f8889dd42d1993aac45ee038fbc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:42:49.823861 29801 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48999f8889dd42d1993aac45ee038fbc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:42:49.823967 29801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48999f8889dd42d1993aac45ee038fbc [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:49.824812 29801 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48999f8889dd42d1993aac45ee038fbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48999f8889dd42d1993aac45ee038fbc" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 38995 } }
I20260926 16:42:49.825160 29801 leader_election.cc:302] T 00000000000000000000000000000000 P 48999f8889dd42d1993aac45ee038fbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48999f8889dd42d1993aac45ee038fbc; no voters:
I20260926 16:42:49.825495 29801 leader_election.cc:288] T 00000000000000000000000000000000 P 48999f8889dd42d1993aac45ee038fbc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:42:49.825590 29806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48999f8889dd42d1993aac45ee038fbc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:49.825838 29806 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48999f8889dd42d1993aac45ee038fbc [term 1 LEADER]: Becoming Leader. State: Replica: 48999f8889dd42d1993aac45ee038fbc, State: Running, Role: LEADER
I20260926 16:42:49.826152 29806 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48999f8889dd42d1993aac45ee038fbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48999f8889dd42d1993aac45ee038fbc" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 38995 } }
I20260926 16:42:49.826784 29801 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48999f8889dd42d1993aac45ee038fbc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:42:49.827334 29808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48999f8889dd42d1993aac45ee038fbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48999f8889dd42d1993aac45ee038fbc. Latest consensus state: current_term: 1 leader_uuid: "48999f8889dd42d1993aac45ee038fbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48999f8889dd42d1993aac45ee038fbc" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 38995 } } }
I20260926 16:42:49.827461 29808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48999f8889dd42d1993aac45ee038fbc [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:49.827732 29807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48999f8889dd42d1993aac45ee038fbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48999f8889dd42d1993aac45ee038fbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48999f8889dd42d1993aac45ee038fbc" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 38995 } } }
I20260926 16:42:49.827818 29807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48999f8889dd42d1993aac45ee038fbc [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:49.828137 29813 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:42:49.828298 29804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:49.814743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40856 (local address 127.21.12.126:38995)
0926 16:42:49.815165 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:49.815181 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:49.815216 (+ 35us) server_negotiation.cc:407] Connection header received
0926 16:42:49.815896 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:49.815914 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:49.816239 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:49.816553 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:49.817376 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:49.818432 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:49.819586 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:49.820039 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:49.820156 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:49.822961 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:49.822966 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:49.822978 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:49.823026 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:49.824889 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:49.825396 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:49.825401 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:49.825405 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:49.825586 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:49.825984 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:49.825986 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:49.825989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:49.826480 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:49.826671 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:49.826966 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 16:42:49.827268 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":126,"threads_started":1}
I20260926 16:42:49.831259 29813 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:42:49.837149 29813 catalog_manager.cc:1347] Generated new cluster ID: a571ce57112b498d8327b8090c99e750
I20260926 16:42:49.837244 29813 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:42:49.859249 29813 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:42:49.860484 29813 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:42:49.869309 29813 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48999f8889dd42d1993aac45ee038fbc: Generated new TSK 0
I20260926 16:42:49.870132 29813 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab.
Entry for principal kudu/127.21.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab.
I20260926 16:42:49.900269 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.65:0
--local_ip_for_outbound_sockets=127.21.12.65
--webserver_interface=127.21.12.65
--webserver_port=0
--tserver_master_addrs=127.21.12.126:38995
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab
--principal=kudu/127.21.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:33869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:50.008066 29827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:50.008323 29827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:50.008365 29827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:50.012297 29827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:50.012351 29827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:50.012445 29827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.65
I20260926 16:42:50.017494 29827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:33869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab
--principal=kudu/127.21.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.65
--webserver_port=0
--tserver_master_addrs=127.21.12.126:38995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.65
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:50.018556 29827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:50.019548 29827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:50.028899 29833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:50.028906 29835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:50.028908 29832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:50.029150 29827 server_base.cc:1034] running on GCE node
I20260926 16:42:50.029702 29827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:50.030333 29827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:50.031519 29827 hybrid_clock.cc:648] HybridClock initialized: now 1790440970031507 us; error 34 us; skew 500 ppm
Sep 26 16:42:50 dist-test-slave-n1hw krb5kdc[29720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440970, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:50.038609 29827 init.cc:377] Logged in from keytab as kudu/127.21.12.65@KRBTEST.COM (short username kudu)
I20260926 16:42:50.039454 29827 webserver.cc:460] Webserver started at http://127.21.12.65:40793/ using document root <none> and password file <none>
I20260926 16:42:50.040004 29827 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:50.040092 29827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:50.040325 29827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:50.042348 29827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-0/data/instance:
uuid: "ab3a7c4ad83c4e66bd5e877a5725572c"
format_stamp: "Formatted at 2026-09-26 16:42:50 on dist-test-slave-n1hw"
server_key: "c04e002768a19affce0f798c46821d38"
server_key_iv: "566ac68c42637531666ece9bab8f752f"
server_key_version: "encryptionkey@0"
I20260926 16:42:50.042892 29827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance:
uuid: "ab3a7c4ad83c4e66bd5e877a5725572c"
format_stamp: "Formatted at 2026-09-26 16:42:50 on dist-test-slave-n1hw"
server_key: "c04e002768a19affce0f798c46821d38"
server_key_iv: "566ac68c42637531666ece9bab8f752f"
server_key_version: "encryptionkey@0"
I20260926 16:42:50.046635 29827 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:42:50.049190 29842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:50.050539 29827 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:42:50.050686 29827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-0/wal
uuid: "ab3a7c4ad83c4e66bd5e877a5725572c"
format_stamp: "Formatted at 2026-09-26 16:42:50 on dist-test-slave-n1hw"
server_key: "c04e002768a19affce0f798c46821d38"
server_key_iv: "566ac68c42637531666ece9bab8f752f"
server_key_version: "encryptionkey@0"
I20260926 16:42:50.050814 29827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:50.087759 29827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:50.091014 29827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:50.091246 29827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:50.091867 29827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:42:50.092729 29827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:50.092803 29827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:50.092885 29827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:50.092933 29827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:50.102291 29827 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.65:33943
I20260926 16:42:50.102325 29955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.65:33943 every 8 connection(s)
I20260926 16:42:50.103214 29827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
I20260926 16:42:50.106690 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 29827
I20260926 16:42:50.106841 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance
I20260926 16:42:50.107163 21553 external_mini_cluster.cc:1428] Setting key ea642a0d428bb0d5e42553a66ca83712
Sep 26 16:42:50 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440970, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.12.66@KRBTEST.COM; defaulting to no policy
I20260926 16:42:50.118052 29804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:50.105059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:60477 (local address 127.21.12.126:38995)
0926 16:42:50.105201 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:50.105205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:50.106356 (+ 1151us) server_negotiation.cc:407] Connection header received
0926 16:42:50.107459 (+ 1103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:50.107463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:50.107522 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:50.107621 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:50.108830 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:50.109410 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:50.110685 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:50.110965 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:50.111098 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:50.114104 (+ 3006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:50.114107 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:50.114110 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:50.114132 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:50.115778 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:50.116346 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:50.116350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:50.116352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:50.116439 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:50.116823 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:50.116826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:50.116828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:50.117064 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:50.117182 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:50.117704 (+ 522us) server_negotiation.cc:299] Negotiation successful
0926 16:42:50.117858 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 16:42:50.118837 29958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:50.105457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.65:60477)
0926 16:42:50.106194 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:50.106238 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:50.107207 (+ 969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:50.107803 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:50.107810 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:50.108179 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:50.108669 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:50.108683 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:50.109575 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:50.109579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:50.110535 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:50.110545 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:50.111132 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:50.111136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:50.111314 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:50.111931 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:50.111948 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:50.113925 (+ 1977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:50.115925 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:50.115931 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:50.115945 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:50.116228 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:50.116548 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:50.116552 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:50.116554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:50.116710 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:50.117179 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:50.117186 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:50.117432 (+ 246us) client_negotiation.cc:769] Sending connection context
0926 16:42:50.117638 (+ 206us) client_negotiation.cc:240] Negotiation successful
0926 16:42:50.117990 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":118,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.66@KRBTEST.COM" created.
I20260926 16:42:50.120112 29956 heartbeater.cc:347] Connected to a master server at 127.21.12.126:38995
I20260926 16:42:50.120378 29956 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:50.120872 29956 heartbeater.cc:511] Master 127.21.12.126:38995 requested a full tablet report, sending...
I20260926 16:42:50.122545 29760 ts_manager.cc:194] Registered new tserver with Master: ab3a7c4ad83c4e66bd5e877a5725572c (127.21.12.65:33943)
I20260926 16:42:50.124193 29760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.65@KRBTEST.COM'} at 127.21.12.65:60477
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab.
Entry for principal kudu/127.21.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab.
I20260926 16:42:50.136984 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.66:0
--local_ip_for_outbound_sockets=127.21.12.66
--webserver_interface=127.21.12.66
--webserver_port=0
--tserver_master_addrs=127.21.12.126:38995
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab
--principal=kudu/127.21.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:33869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:50.244436 29961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:50.244690 29961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:50.244732 29961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:50.248597 29961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:50.248646 29961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:50.248705 29961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.66
I20260926 16:42:50.254034 29961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:33869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab
--principal=kudu/127.21.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.66
--webserver_port=0
--tserver_master_addrs=127.21.12.126:38995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.66
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:50.255018 29961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:50.255990 29961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:50.265552 29969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:50.265564 29967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:50.265678 29961 server_base.cc:1034] running on GCE node
W20260926 16:42:50.265651 29966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:50.266206 29961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:50.266884 29961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:50.268088 29961 hybrid_clock.cc:648] HybridClock initialized: now 1790440970268077 us; error 29 us; skew 500 ppm
Sep 26 16:42:50 dist-test-slave-n1hw krb5kdc[29720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440970, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:50.270999 29961 init.cc:377] Logged in from keytab as kudu/127.21.12.66@KRBTEST.COM (short username kudu)
I20260926 16:42:50.271786 29961 webserver.cc:460] Webserver started at http://127.21.12.66:46565/ using document root <none> and password file <none>
I20260926 16:42:50.272300 29961 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:50.272357 29961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:50.272526 29961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:50.274554 29961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-1/data/instance:
uuid: "625e9206c0b6470c8f48d8379c497114"
format_stamp: "Formatted at 2026-09-26 16:42:50 on dist-test-slave-n1hw"
server_key: "e1762acda715379439d2aaec8c4de085"
server_key_iv: "4be8f74c570a07e3dbeb0070523e90b9"
server_key_version: "encryptionkey@0"
I20260926 16:42:50.275066 29961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance:
uuid: "625e9206c0b6470c8f48d8379c497114"
format_stamp: "Formatted at 2026-09-26 16:42:50 on dist-test-slave-n1hw"
server_key: "e1762acda715379439d2aaec8c4de085"
server_key_iv: "4be8f74c570a07e3dbeb0070523e90b9"
server_key_version: "encryptionkey@0"
I20260926 16:42:50.278615 29961 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:42:50.280906 29976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:50.282245 29961 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:42:50.282364 29961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-1/wal
uuid: "625e9206c0b6470c8f48d8379c497114"
format_stamp: "Formatted at 2026-09-26 16:42:50 on dist-test-slave-n1hw"
server_key: "e1762acda715379439d2aaec8c4de085"
server_key_iv: "4be8f74c570a07e3dbeb0070523e90b9"
server_key_version: "encryptionkey@0"
I20260926 16:42:50.282469 29961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:50.304025 29961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:50.308451 29961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:50.308670 29961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:50.309536 29961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:42:50.310750 29961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:50.310809 29961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:50.310871 29961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:50.310902 29961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:50.324388 29961 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.66:38469
I20260926 16:42:50.324419 30089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.66:38469 every 8 connection(s)
I20260926 16:42:50.325524 29961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
I20260926 16:42:50.333292 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 29961
I20260926 16:42:50.333442 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance
I20260926 16:42:50.333717 21553 external_mini_cluster.cc:1428] Setting key cb5c00e78d3f1dbe13f880c6a667caaf
Sep 26 16:42:50 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440970, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.66@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:42:50.341825 29804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:50.327810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:60433 (local address 127.21.12.126:38995)
0926 16:42:50.329590 (+ 1780us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:50.329595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:50.329666 (+ 71us) server_negotiation.cc:407] Connection header received
0926 16:42:50.330829 (+ 1163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:50.330833 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:50.330893 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:50.330939 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:50.332621 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:50.333175 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:50.334885 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:50.335092 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:50.335211 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:50.337871 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:50.337878 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:50.337884 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:50.337916 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:50.339646 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:50.340172 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:50.340181 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:50.340182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:50.340259 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:50.340680 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:50.340684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:50.340686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:50.340872 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:50.340979 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:50.341495 (+ 516us) server_negotiation.cc:299] Negotiation successful
0926 16:42:50.341621 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1674}
I20260926 16:42:50.342631 30092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:50.328433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.66:60433)
0926 16:42:50.329371 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:50.329439 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:50.330569 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:50.331303 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:50.331312 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:50.331772 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:50.332460 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:50.332476 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:50.333303 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:50.333308 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:50.334371 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:50.334380 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:50.335257 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:50.335261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:50.335434 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:50.336069 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:50.336085 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:50.337664 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:50.339769 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:50.339774 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:50.339786 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:50.340046 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:50.340371 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:50.340379 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:50.340381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:50.340532 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:50.340990 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:50.340999 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:50.341244 (+ 245us) client_negotiation.cc:769] Sending connection context
0926 16:42:50.341438 (+ 194us) client_negotiation.cc:240] Negotiation successful
0926 16:42:50.341807 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":156,"threads_started":1}
WARNING: no policy specified for kudu/127.21.12.67@KRBTEST.COM; defaulting to no policy
I20260926 16:42:50.343900 30090 heartbeater.cc:347] Connected to a master server at 127.21.12.126:38995
I20260926 16:42:50.344118 30090 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:50.344662 30090 heartbeater.cc:511] Master 127.21.12.126:38995 requested a full tablet report, sending...
I20260926 16:42:50.345664 29760 ts_manager.cc:194] Registered new tserver with Master: 625e9206c0b6470c8f48d8379c497114 (127.21.12.66:38469)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.67@KRBTEST.COM" created.
I20260926 16:42:50.346366 29760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.66@KRBTEST.COM'} at 127.21.12.66:60433
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab.
Entry for principal kudu/127.21.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab.
I20260926 16:42:50.360975 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.67:0
--local_ip_for_outbound_sockets=127.21.12.67
--webserver_interface=127.21.12.67
--webserver_port=0
--tserver_master_addrs=127.21.12.126:38995
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab
--principal=kudu/127.21.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:33869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:50.468645 30095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:50.468887 30095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:50.468928 30095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:50.472800 30095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:50.472851 30095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:50.472908 30095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.67
I20260926 16:42:50.477882 30095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:33869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab
--principal=kudu/127.21.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.67
--webserver_port=0
--tserver_master_addrs=127.21.12.126:38995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.67
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:50.478798 30095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:50.479720 30095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:50.489526 30103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:50.489588 30100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:50.489523 30101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:50.489979 30095 server_base.cc:1034] running on GCE node
I20260926 16:42:50.490445 30095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:50.491098 30095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:50.492313 30095 hybrid_clock.cc:648] HybridClock initialized: now 1790440970492282 us; error 42 us; skew 500 ppm
Sep 26 16:42:50 dist-test-slave-n1hw krb5kdc[29720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440970, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:50.495553 30095 init.cc:377] Logged in from keytab as kudu/127.21.12.67@KRBTEST.COM (short username kudu)
I20260926 16:42:50.496560 30095 webserver.cc:460] Webserver started at http://127.21.12.67:37873/ using document root <none> and password file <none>
I20260926 16:42:50.497144 30095 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:50.497234 30095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:50.497465 30095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:50.499609 30095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-2/data/instance:
uuid: "177f61a0bea341af80978976095347b8"
format_stamp: "Formatted at 2026-09-26 16:42:50 on dist-test-slave-n1hw"
server_key: "b415b9db4083f38c336e8c5f57e3390e"
server_key_iv: "95b080db63551744daadb429952436ce"
server_key_version: "encryptionkey@0"
I20260926 16:42:50.500202 30095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance:
uuid: "177f61a0bea341af80978976095347b8"
format_stamp: "Formatted at 2026-09-26 16:42:50 on dist-test-slave-n1hw"
server_key: "b415b9db4083f38c336e8c5f57e3390e"
server_key_iv: "95b080db63551744daadb429952436ce"
server_key_version: "encryptionkey@0"
I20260926 16:42:50.504077 30095 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 16:42:50.506515 30110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:50.507694 30095 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:42:50.507833 30095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-2/wal
uuid: "177f61a0bea341af80978976095347b8"
format_stamp: "Formatted at 2026-09-26 16:42:50 on dist-test-slave-n1hw"
server_key: "b415b9db4083f38c336e8c5f57e3390e"
server_key_iv: "95b080db63551744daadb429952436ce"
server_key_version: "encryptionkey@0"
I20260926 16:42:50.507942 30095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:50.517551 30095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:50.520932 30095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:50.521129 30095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:50.521684 30095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:42:50.522511 30095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:50.522588 30095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:50.522668 30095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:50.522722 30095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:50.532009 30095 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.67:37991
I20260926 16:42:50.532037 30223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.67:37991 every 8 connection(s)
I20260926 16:42:50.532933 30095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
I20260926 16:42:50.536774 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 30095
I20260926 16:42:50.536965 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance
I20260926 16:42:50.537256 21553 external_mini_cluster.cc:1428] Setting key 9e3f93f16aa9d9a61944a6757dc91324
Sep 26 16:42:50 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440970, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.67@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:42:50.546052 29804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:50.534655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:57799 (local address 127.21.12.126:38995)
0926 16:42:50.534799 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:50.534803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:50.535858 (+ 1055us) server_negotiation.cc:407] Connection header received
0926 16:42:50.536764 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:50.536767 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:50.536823 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:50.536936 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:50.538279 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:50.538814 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:50.540048 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:50.540281 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:50.540394 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:50.542766 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:50.542772 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:50.542778 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:50.542803 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:50.544049 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:50.544547 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:50.544552 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:50.544554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:50.544650 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:50.544991 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:50.544994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:50.544996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:50.545180 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:50.545287 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:50.545778 (+ 491us) server_negotiation.cc:299] Negotiation successful
0926 16:42:50.545896 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 16:42:50.546829 30226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:50.535068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.67:57799)
0926 16:42:50.535704 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:50.535743 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:50.536518 (+ 775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:50.537114 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:50.537120 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:50.537529 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:50.538139 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:50.538156 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:50.538937 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:50.538941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:50.539921 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:50.539931 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:50.540425 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:50.540429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:50.540598 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:50.541112 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:50.541127 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:50.542578 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:50.544170 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:50.544177 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:50.544191 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:50.544440 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:50.544742 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:50.544747 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:50.544750 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:50.544885 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:50.545284 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:50.545290 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:50.545520 (+ 230us) client_negotiation.cc:769] Sending connection context
0926 16:42:50.545724 (+ 204us) client_negotiation.cc:240] Negotiation successful
0926 16:42:50.546054 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":106,"threads_started":1}
I20260926 16:42:50.547855 30224 heartbeater.cc:347] Connected to a master server at 127.21.12.126:38995
I20260926 16:42:50.548058 30224 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:50.548481 30224 heartbeater.cc:511] Master 127.21.12.126:38995 requested a full tablet report, sending...
I20260926 16:42:50.549604 29760 ts_manager.cc:194] Registered new tserver with Master: 177f61a0bea341af80978976095347b8 (127.21.12.67:37991)
I20260926 16:42:50.550390 29760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.67@KRBTEST.COM'} at 127.21.12.67:57799
I20260926 16:42:50.551096 21553 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.21.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.65.keytab.
Entry for principal oryx/127.21.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.65.keytab.
WARNING: no policy specified for oryx/127.21.12.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.66.keytab.
Entry for principal oryx/127.21.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.66.keytab.
WARNING: no policy specified for oryx/127.21.12.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.67.keytab.
Entry for principal oryx/127.21.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.67.keytab.
I20260926 16:42:50.629058 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 29827
I20260926 16:42:50.634948 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 29961
I20260926 16:42:50.640040 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 30095
I20260926 16:42:50.645283 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 29733
I20260926 16:42:50.651458 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:38995
--webserver_interface=127.21.12.126
--webserver_port=40883
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:33869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:38995
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:50.760692 30233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:50.760926 30233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:50.760964 30233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:50.764889 30233 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:42:50.764940 30233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:50.764961 30233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:50.764976 30233 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:42:50.765069 30233 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:42:50.770358 30233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:33869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:38995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:38995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.126
--webserver_port=40883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:50.771348 30233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:50.772329 30233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:50.780609 30238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:50.780603 30239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:50.780601 30241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:50.780799 30233 server_base.cc:1034] running on GCE node
I20260926 16:42:50.781373 30233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:50.782648 30233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:50.783841 30233 hybrid_clock.cc:648] HybridClock initialized: now 1790440970783820 us; error 31 us; skew 500 ppm
Sep 26 16:42:50 dist-test-slave-n1hw krb5kdc[29720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440970, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:50.786825 30233 init.cc:377] Logged in from keytab as kudu/127.21.12.126@KRBTEST.COM (short username kudu)
I20260926 16:42:50.787729 30233 webserver.cc:460] Webserver started at http://127.21.12.126:40883/ using document root <none> and password file <none>
I20260926 16:42:50.788271 30233 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:50.788372 30233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:50.792116 30233 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260926 16:42:50.794453 30248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:50.795785 30233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:42:50.795929 30233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "48999f8889dd42d1993aac45ee038fbc"
format_stamp: "Formatted at 2026-09-26 16:42:49 on dist-test-slave-n1hw"
server_key: "24fb9cd80f17063f05d1a0222c41f2e3"
server_key_iv: "67e320f97dd9f1e40072655f80f7ca66"
server_key_version: "encryptionkey@0"
I20260926 16:42:50.796509 30233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:50.807686 30233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:50.810689 30233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:50.810874 30233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:50.817811 30233 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.126:38995
I20260926 16:42:50.817833 30300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.126:38995 every 8 connection(s)
I20260926 16:42:50.818871 30233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:42:50.822911 30301 sys_catalog.cc:263] Verifying existing consensus state
I20260926 16:42:50.824468 30301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48999f8889dd42d1993aac45ee038fbc: Bootstrap starting.
I20260926 16:42:50.827363 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 30233
I20260926 16:42:50.827821 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.65:33943
--local_ip_for_outbound_sockets=127.21.12.65
--tserver_master_addrs=127.21.12.126:38995
--webserver_port=40793
--webserver_interface=127.21.12.65
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab
--principal=kudu/127.21.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:33869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.12.65
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:42:50.840889 30301 log.cc:824] T 00000000000000000000000000000000 P 48999f8889dd42d1993aac45ee038fbc: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:50.847441 30301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48999f8889dd42d1993aac45ee038fbc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 16:42:50.847806 30301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48999f8889dd42d1993aac45ee038fbc: Bootstrap complete.
I20260926 16:42:50.852015 30301 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48999f8889dd42d1993aac45ee038fbc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48999f8889dd42d1993aac45ee038fbc" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 38995 } }
I20260926 16:42:50.852442 30301 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48999f8889dd42d1993aac45ee038fbc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48999f8889dd42d1993aac45ee038fbc, State: Initialized, Role: FOLLOWER
I20260926 16:42:50.852968 30301 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48999f8889dd42d1993aac45ee038fbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48999f8889dd42d1993aac45ee038fbc" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 38995 } }
I20260926 16:42:50.853113 30301 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48999f8889dd42d1993aac45ee038fbc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:42:50.853201 30301 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48999f8889dd42d1993aac45ee038fbc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:42:50.853317 30301 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48999f8889dd42d1993aac45ee038fbc [term 1 FOLLOWER]: Advancing to term 2
I20260926 16:42:50.855234 30301 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48999f8889dd42d1993aac45ee038fbc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48999f8889dd42d1993aac45ee038fbc" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 38995 } }
I20260926 16:42:50.855554 30301 leader_election.cc:302] T 00000000000000000000000000000000 P 48999f8889dd42d1993aac45ee038fbc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48999f8889dd42d1993aac45ee038fbc; no voters:
I20260926 16:42:50.855877 30301 leader_election.cc:288] T 00000000000000000000000000000000 P 48999f8889dd42d1993aac45ee038fbc [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 16:42:50.855962 30305 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48999f8889dd42d1993aac45ee038fbc [term 2 FOLLOWER]: Leader election won for term 2
I20260926 16:42:50.856174 30305 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48999f8889dd42d1993aac45ee038fbc [term 2 LEADER]: Becoming Leader. State: Replica: 48999f8889dd42d1993aac45ee038fbc, State: Running, Role: LEADER
I20260926 16:42:50.856456 30305 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48999f8889dd42d1993aac45ee038fbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48999f8889dd42d1993aac45ee038fbc" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 38995 } }
I20260926 16:42:50.857110 30301 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48999f8889dd42d1993aac45ee038fbc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:42:50.857897 30307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48999f8889dd42d1993aac45ee038fbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "48999f8889dd42d1993aac45ee038fbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48999f8889dd42d1993aac45ee038fbc" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 38995 } } }
I20260926 16:42:50.857931 30306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48999f8889dd42d1993aac45ee038fbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48999f8889dd42d1993aac45ee038fbc. Latest consensus state: current_term: 2 leader_uuid: "48999f8889dd42d1993aac45ee038fbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48999f8889dd42d1993aac45ee038fbc" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 38995 } } }
I20260926 16:42:50.858022 30307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48999f8889dd42d1993aac45ee038fbc [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:50.858037 30306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48999f8889dd42d1993aac45ee038fbc [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:50.858386 30309 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:42:50.861853 30309 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:42:50.862463 30309 catalog_manager.cc:1259] Loaded cluster ID: a571ce57112b498d8327b8090c99e750
I20260926 16:42:50.862521 30309 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:42:50.864791 30309 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:42:50.865429 30309 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 48999f8889dd42d1993aac45ee038fbc: Loaded TSK: 0
I20260926 16:42:50.866626 30309 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 16:42:50.940887 30303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:50.941164 30303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:50.941232 30303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:50.945325 30303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:50.945411 30303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:50.945501 30303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.65
I20260926 16:42:50.950845 30303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:33869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.65.keytab
--principal=oryx/127.21.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.65:33943
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.65
--webserver_port=40793
--tserver_master_addrs=127.21.12.126:38995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.65
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:50.951836 30303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:50.952839 30303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:50.963467 30327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:50.963474 30330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:50.963474 30328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:50.963961 30303 server_base.cc:1034] running on GCE node
I20260926 16:42:50.964455 30303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:50.965248 30303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:50.966450 30303 hybrid_clock.cc:648] HybridClock initialized: now 1790440970966425 us; error 34 us; skew 500 ppm
Sep 26 16:42:50 dist-test-slave-n1hw krb5kdc[29720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440970, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:50.969937 30303 init.cc:377] Logged in from keytab as oryx/127.21.12.65@KRBTEST.COM (short username oryx)
I20260926 16:42:50.971047 30303 webserver.cc:460] Webserver started at http://127.21.12.65:40793/ using document root <none> and password file <none>
I20260926 16:42:50.971776 30303 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:50.971889 30303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:50.976939 30303 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 16:42:50.979449 30337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:50.980759 30303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 16:42:50.980899 30303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-0/wal
uuid: "ab3a7c4ad83c4e66bd5e877a5725572c"
format_stamp: "Formatted at 2026-09-26 16:42:50 on dist-test-slave-n1hw"
server_key: "c04e002768a19affce0f798c46821d38"
server_key_iv: "566ac68c42637531666ece9bab8f752f"
server_key_version: "encryptionkey@0"
I20260926 16:42:50.981439 30303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:50.994634 30303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:50.998081 30303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:50.998293 30303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:50.998977 30303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:42:50.999894 30303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:50.999969 30303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:51.000039 30303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:51.000087 30303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:51.009109 30303 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.65:33943
I20260926 16:42:51.009130 30450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.65:33943 every 8 connection(s)
I20260926 16:42:51.010165 30303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
I20260926 16:42:51.014006 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 30303
I20260926 16:42:51.014488 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.66:38469
--local_ip_for_outbound_sockets=127.21.12.66
--tserver_master_addrs=127.21.12.126:38995
--webserver_port=46565
--webserver_interface=127.21.12.66
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab
--principal=kudu/127.21.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:33869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.12.66
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 16:42:51 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.65@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:42:51 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.65@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
W20260926 16:42:51.023253 30453 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:51.012190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:51721 (local address 127.21.12.126:38995)
0926 16:42:51.012636 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:51.012655 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:51.013127 (+ 472us) server_negotiation.cc:407] Connection header received
0926 16:42:51.014238 (+ 1111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:51.014255 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:51.014671 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:51.015044 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:51.016090 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:51.017117 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:51.018313 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:51.018789 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:51.018964 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:51.022542 (+ 3578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.65:51721: BlockingRecv error: recv got EOF from 127.21.12.65:51721 (error 108)
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":92,"threads_started":1}
I20260926 16:42:51.023260 30454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:51.012236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.65:51721)
0926 16:42:51.012967 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:51.013006 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:51.013742 (+ 736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:51.015101 (+ 1359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:51.015108 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:51.015444 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:51.015896 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:51.015909 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:51.017269 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:51.017278 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:51.018185 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:51.018195 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:51.018947 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:51.018951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:51.019135 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:51.019794 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:51.019810 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:51.022328 (+ 2518us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":152,"threads_started":1}
W20260926 16:42:51.023957 30451 heartbeater.cc:644] Failed to heartbeat to 127.21.12.126:38995 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.12.126:38995: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Sep 26 16:42:51 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.65@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:42:51 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.65@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:42:51.029987 30454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:51.024460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.65:46505)
0926 16:42:51.024597 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:51.024611 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:51.024696 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:51.024975 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:51.024977 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:51.025127 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:51.025232 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:51.025249 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:51.026300 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:51.026302 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:51.026848 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:51.026856 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:51.027418 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:51.027420 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:51.027495 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:51.028039 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:51.028052 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:51.029896 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":67}
W20260926 16:42:51.030359 30453 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:51.024565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:46505 (local address 127.21.12.126:38995)
0926 16:42:51.024720 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:51.024727 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:51.024743 (+ 16us) server_negotiation.cc:407] Connection header received
0926 16:42:51.024787 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:51.024792 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:51.024851 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:51.024953 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:51.025355 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:51.026177 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:51.026956 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:51.027298 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:51.027398 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:51.030050 (+ 2652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.65:46505: BlockingRecv error: recv got EOF from 127.21.12.65:46505 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 16:42:51 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.65@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:42:51 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.65@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:42:51.035645 30454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:51.030500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.65:45925)
0926 16:42:51.030645 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:51.030657 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:51.030738 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:51.030976 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:51.030978 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:51.031115 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:51.031180 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:51.031185 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:51.032176 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:51.032179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:51.032790 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:51.032799 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:51.033335 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:51.033337 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:51.033405 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:51.033909 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:51.033920 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:51.035551 (+ 1631us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":88}
W20260926 16:42:51.035844 30451 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 16:42:51.036026 30453 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:51.030628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:45925 (local address 127.21.12.126:38995)
0926 16:42:51.030738 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:51.030744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:51.030760 (+ 16us) server_negotiation.cc:407] Connection header received
0926 16:42:51.030805 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:51.030809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:51.030858 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:51.030946 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:51.031289 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:51.032064 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:51.032897 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:51.033232 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:51.033332 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:51.035720 (+ 2388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.65:45925: BlockingRecv error: recv got EOF from 127.21.12.65:45925 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260926 16:42:51.129444 30455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:51.129683 30455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:51.129760 30455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:51.133903 30455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:51.133960 30455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:51.134017 30455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.66
I20260926 16:42:51.139081 30455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:33869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.66.keytab
--principal=oryx/127.21.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.66:38469
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.66
--webserver_port=46565
--tserver_master_addrs=127.21.12.126:38995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.66
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:51.140058 30455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:51.141031 30455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:51.150206 30461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:51.150244 30460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:51.150388 30455 server_base.cc:1034] running on GCE node
W20260926 16:42:51.150200 30463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:51.150942 30455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:51.151574 30455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:51.152755 30455 hybrid_clock.cc:648] HybridClock initialized: now 1790440971152722 us; error 44 us; skew 500 ppm
Sep 26 16:42:51 dist-test-slave-n1hw krb5kdc[29720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440971, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:51.155556 30455 init.cc:377] Logged in from keytab as oryx/127.21.12.66@KRBTEST.COM (short username oryx)
I20260926 16:42:51.156394 30455 webserver.cc:460] Webserver started at http://127.21.12.66:46565/ using document root <none> and password file <none>
I20260926 16:42:51.157008 30455 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:51.157147 30455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:51.161388 30455 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:42:51.163520 30470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:51.164734 30455 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:42:51.164824 30455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-1/wal
uuid: "625e9206c0b6470c8f48d8379c497114"
format_stamp: "Formatted at 2026-09-26 16:42:50 on dist-test-slave-n1hw"
server_key: "e1762acda715379439d2aaec8c4de085"
server_key_iv: "4be8f74c570a07e3dbeb0070523e90b9"
server_key_version: "encryptionkey@0"
I20260926 16:42:51.165519 30455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:51.188898 30455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:51.192137 30455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:51.192281 30455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:51.192775 30455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:42:51.193493 30455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:51.193538 30455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:51.193578 30455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:51.193593 30455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:51.202383 30455 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.66:38469
I20260926 16:42:51.202455 30583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.66:38469 every 8 connection(s)
I20260926 16:42:51.203403 30455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
I20260926 16:42:51.211109 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 30455
I20260926 16:42:51.211576 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.67:37991
--local_ip_for_outbound_sockets=127.21.12.67
--tserver_master_addrs=127.21.12.126:38995
--webserver_port=37873
--webserver_interface=127.21.12.67
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab
--principal=kudu/127.21.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:33869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.12.67
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 16:42:51 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.66@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:42:51 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.66@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
W20260926 16:42:51.214458 30453 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:51.205177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:46883 (local address 127.21.12.126:38995)
0926 16:42:51.205320 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:51.205325 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:51.206481 (+ 1156us) server_negotiation.cc:407] Connection header received
0926 16:42:51.207291 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:51.207293 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:51.207341 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:51.207415 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:51.208577 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:51.209098 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:51.210279 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:51.210488 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:51.210569 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:51.214359 (+ 3790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.66:46883: BlockingRecv error: recv got EOF from 127.21.12.66:46883 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 16:42:51.215230 30586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:51.205606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.66:46883)
0926 16:42:51.206326 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:51.206367 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:51.207110 (+ 743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:51.207629 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:51.207635 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:51.207972 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:51.208429 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:51.208441 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:51.209216 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:51.209219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:51.210169 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:51.210178 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:51.210624 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:51.210626 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:51.210793 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:51.211444 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:51.211460 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:51.214254 (+ 2794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":163,"threads_started":1}
W20260926 16:42:51.215772 30584 heartbeater.cc:644] Failed to heartbeat to 127.21.12.126:38995 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.12.126:38995: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Sep 26 16:42:51 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.66@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:42:51 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.66@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:42:51.222128 30586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:51.216233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.66:60399)
0926 16:42:51.216421 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:51.216436 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:51.216527 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:51.216809 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:51.216815 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:51.216994 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:51.217129 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:51.217139 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:51.217916 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:51.217921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:51.218945 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:51.218960 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:51.219377 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:51.219382 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:51.219508 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:51.219965 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:51.219984 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:51.221969 (+ 1985us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105}
W20260926 16:42:51.222170 30453 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:51.216313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:60399 (local address 127.21.12.126:38995)
0926 16:42:51.216422 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:51.216425 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:51.216482 (+ 57us) server_negotiation.cc:407] Connection header received
0926 16:42:51.216625 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:51.216627 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:51.216693 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:51.216760 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:51.217253 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:51.217798 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:51.219067 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:51.219266 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:51.219339 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:51.222072 (+ 2733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.66:60399: BlockingRecv error: recv got EOF from 127.21.12.66:60399 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 26 16:42:51 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.66@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:42:51 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.66@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:42:51.227852 30586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:51.222774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.66:58177)
0926 16:42:51.222954 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:51.222970 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:51.223060 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:51.223291 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:51.223294 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:51.223460 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:51.223542 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:51.223548 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:51.224230 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:51.224233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:51.225079 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:51.225087 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:51.225480 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:51.225485 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:51.225567 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:51.226004 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:51.226017 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:51.227745 (+ 1728us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99}
W20260926 16:42:51.227959 30453 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:51.222845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:58177 (local address 127.21.12.126:38995)
0926 16:42:51.222956 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:51.222959 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:51.223016 (+ 57us) server_negotiation.cc:407] Connection header received
0926 16:42:51.223144 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:51.223146 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:51.223181 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:51.223244 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:51.223681 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:51.224128 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:51.225190 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:51.225381 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:51.225455 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:51.227851 (+ 2396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.66:58177: BlockingRecv error: recv got EOF from 127.21.12.66:58177 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260926 16:42:51.228127 30584 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 16:42:51.323261 30587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:51.323494 30587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:51.323541 30587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:51.327508 30587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:51.327564 30587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:51.327661 30587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.67
I20260926 16:42:51.332697 30587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:33869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/krb5kdc/oryx_127.21.12.67.keytab
--principal=oryx/127.21.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.67:37991
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.67
--webserver_port=37873
--tserver_master_addrs=127.21.12.126:38995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.67
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:42:51.333786 30587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:51.334702 30587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:51.343961 30592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:51.343940 30587 server_base.cc:1034] running on GCE node
W20260926 16:42:51.343914 30593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:51.343914 30595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:51.344511 30587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:51.345108 30587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:51.346278 30587 hybrid_clock.cc:648] HybridClock initialized: now 1790440971346262 us; error 22 us; skew 500 ppm
Sep 26 16:42:51 dist-test-slave-n1hw krb5kdc[29720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440971, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:51.349006 30587 init.cc:377] Logged in from keytab as oryx/127.21.12.67@KRBTEST.COM (short username oryx)
I20260926 16:42:51.349931 30587 webserver.cc:460] Webserver started at http://127.21.12.67:37873/ using document root <none> and password file <none>
I20260926 16:42:51.350472 30587 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:51.350539 30587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:51.354128 30587 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260926 16:42:51.356032 30602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:51.357167 30587 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:42:51.357275 30587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-2/wal
uuid: "177f61a0bea341af80978976095347b8"
format_stamp: "Formatted at 2026-09-26 16:42:50 on dist-test-slave-n1hw"
server_key: "b415b9db4083f38c336e8c5f57e3390e"
server_key_iv: "95b080db63551744daadb429952436ce"
server_key_version: "encryptionkey@0"
I20260926 16:42:51.357851 30587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:51.375510 30587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:51.378451 30587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:51.378638 30587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:51.379166 30587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:42:51.379925 30587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:51.379992 30587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:51.380060 30587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:51.380119 30587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:51.389082 30587 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.67:37991
I20260926 16:42:51.389103 30715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.67:37991 every 8 connection(s)
I20260926 16:42:51.390118 30587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
I20260926 16:42:51.398293 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 30587
Sep 26 16:42:51 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.67@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:42:51 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.67@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
W20260926 16:42:51.400502 30453 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:51.391811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:44777 (local address 127.21.12.126:38995)
0926 16:42:51.391945 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:51.391948 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:51.392951 (+ 1003us) server_negotiation.cc:407] Connection header received
0926 16:42:51.393868 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:51.393871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:51.393926 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:51.394008 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:51.395139 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:51.395668 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:51.396786 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:51.397019 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:51.397132 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:51.400385 (+ 3253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.67:44777: BlockingRecv error: recv got EOF from 127.21.12.67:44777 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 16:42:51.401119 30718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:51.392196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.67:44777)
0926 16:42:51.392808 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:51.392848 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:51.393613 (+ 765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:51.394198 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:51.394205 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:51.394540 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:51.394990 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:51.395003 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:51.395793 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:51.395797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:51.396662 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:51.396671 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:51.397179 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:51.397183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:51.397352 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:51.397887 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:51.397902 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:51.400288 (+ 2386us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":105,"threads_started":1}
W20260926 16:42:51.401536 30716 heartbeater.cc:644] Failed to heartbeat to 127.21.12.126:38995 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.12.126:38995: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Sep 26 16:42:51 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.67@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:42:51 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.67@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:42:51.406802 30718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:51.401975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.67:56877)
0926 16:42:51.402055 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:51.402076 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:51.402134 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:51.402430 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:51.402433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:51.402573 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:51.402652 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:51.402656 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:51.403349 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:51.403355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:51.403874 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:51.403881 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:51.404333 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:51.404337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:51.404428 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:51.404926 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:51.404958 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:51.406675 (+ 1717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":24}
W20260926 16:42:51.406883 30453 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:51.402137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:56877 (local address 127.21.12.126:38995)
0926 16:42:51.402253 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:51.402257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:51.402268 (+ 11us) server_negotiation.cc:407] Connection header received
0926 16:42:51.402303 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:51.402304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:51.402334 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:51.402415 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:51.402759 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:51.403247 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:51.403987 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:51.404214 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:51.404320 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:51.406773 (+ 2453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.67:56877: BlockingRecv error: recv got EOF from 127.21.12.67:56877 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 16:42:51.408181 30720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:51.399312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40860 (local address 127.21.12.126:38995)
0926 16:42:51.399600 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:51.399605 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:51.399617 (+ 12us) server_negotiation.cc:407] Connection header received
0926 16:42:51.399654 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:51.399656 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:51.399690 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:51.399782 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:51.400494 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:51.401314 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:51.402475 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:51.402717 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:51.402824 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:51.404159 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:51.404165 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:51.404189 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:51.404213 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:51.405894 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:51.406315 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:51.406322 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:51.406329 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:51.406461 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:51.406755 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:51.406758 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:51.406760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:51.407215 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:51.407403 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:51.407690 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 16:42:51.408016 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":96,"threads_started":1}
Sep 26 16:42:51 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.67@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:42:51 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.67@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:42:51.412102 30718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:51.407235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.67:40403)
0926 16:42:51.407329 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:51.407343 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:51.407405 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:51.407781 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:51.407784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:51.407911 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:51.407974 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:51.407978 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:51.408723 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:51.408726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:51.409352 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:51.409361 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:51.409930 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:51.409937 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:51.410016 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:51.410433 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:51.410445 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:51.412024 (+ 1579us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260926 16:42:51.412226 30453 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:51.407438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:40403 (local address 127.21.12.126:38995)
0926 16:42:51.407589 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:51.407592 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:51.407602 (+ 10us) server_negotiation.cc:407] Connection header received
0926 16:42:51.407629 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:51.407631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:51.407663 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:51.407746 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:51.408078 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:51.408625 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:51.409470 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:51.409811 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:51.409918 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:51.412129 (+ 2211us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.67:40403: BlockingRecv error: recv got EOF from 127.21.12.67:40403 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260926 16:42:51.412302 30716 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 16:42:52 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.65@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:42:52 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.65@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:42:52.041627 30721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:52.036449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.65:35981)
0926 16:42:52.036711 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:52.036725 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:52.036814 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:52.037016 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:52.037018 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:52.037183 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:52.037300 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:52.037305 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:52.038236 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:52.038239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:52.038782 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:52.038789 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:52.039248 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:52.039250 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:52.039309 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:52.039765 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:52.039776 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:52.041513 (+ 1737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":95,"threads_started":1}
W20260926 16:42:52.041882 30722 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:52.036510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:35981 (local address 127.21.12.126:38995)
0926 16:42:52.036738 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:52.036741 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:52.036782 (+ 41us) server_negotiation.cc:407] Connection header received
0926 16:42:52.036888 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:52.036890 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:52.036931 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:52.037007 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:52.037433 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:52.038129 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:52.038886 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:52.039138 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:52.039224 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:52.041690 (+ 2466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.65:35981: BlockingRecv error: recv got EOF from 127.21.12.65:35981 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":95,"threads_started":1}
Sep 26 16:42:52 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.66@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:42:52 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.66@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:42:52.233978 30723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:52.228817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.66:54117)
0926 16:42:52.229052 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:52.229068 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:52.229203 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:52.229556 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:52.229560 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:52.229786 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:52.229906 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:52.229913 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:52.230615 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:52.230617 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:52.231184 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:52.231191 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:52.231570 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:52.231572 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:52.231641 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:52.232109 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:52.232120 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:52.233863 (+ 1743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":73,"threads_started":1}
W20260926 16:42:52.234130 30722 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:52.228874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:54117 (local address 127.21.12.126:38995)
0926 16:42:52.229036 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:52.229040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:52.229151 (+ 111us) server_negotiation.cc:407] Connection header received
0926 16:42:52.229316 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:52.229322 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:52.229375 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:52.229485 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:52.230011 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:52.230523 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:52.231277 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:52.231480 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:52.231551 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:52.233980 (+ 2429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.66:54117: BlockingRecv error: recv got EOF from 127.21.12.66:54117 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 26 16:42:52 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.67@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:42:52 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.67@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:42:52.418735 30724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:52.412966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.67:45171)
0926 16:42:52.413270 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:52.413283 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:52.413425 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:52.413700 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:52.413704 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:52.413939 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:52.414057 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:52.414063 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:52.414827 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:52.414833 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:52.415417 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:52.415425 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:52.415890 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:52.415894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:52.415995 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:52.416530 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:52.416542 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:52.418562 (+ 2020us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":93,"threads_started":1}
W20260926 16:42:52.418834 30722 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:52.413035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:45171 (local address 127.21.12.126:38995)
0926 16:42:52.413220 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:52.413224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:52.413364 (+ 140us) server_negotiation.cc:407] Connection header received
0926 16:42:52.413529 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:52.413532 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:52.413587 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:52.413707 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:52.414181 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:52.414715 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:52.415542 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:52.415776 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:52.415886 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:52.418714 (+ 2828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.67:45171: BlockingRecv error: recv got EOF from 127.21.12.67:45171 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 26 16:42:53 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.65@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:42:53 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.65@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:42:53.047726 30725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:53.042388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.65:34567)
0926 16:42:53.042681 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:53.042699 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:53.042799 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:53.043044 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:53.043047 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:53.043239 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:53.043335 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:53.043341 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:53.044069 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:53.044071 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:53.044641 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:53.044651 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:53.045050 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:53.045052 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:53.045113 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:53.045719 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:53.045731 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:53.047582 (+ 1851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":68,"threads_started":1}
W20260926 16:42:53.047917 30726 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:53.042454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:34567 (local address 127.21.12.126:38995)
0926 16:42:53.042807 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:53.042811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:53.042823 (+ 12us) server_negotiation.cc:407] Connection header received
0926 16:42:53.042860 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:53.042862 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:53.042901 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:53.042975 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:53.043441 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:53.043973 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:53.044745 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:53.044959 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:53.045031 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:53.047737 (+ 2706us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.65:34567: BlockingRecv error: recv got EOF from 127.21.12.65:34567 (error 108)
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":177,"threads_started":1}
Sep 26 16:42:53 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.66@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:42:53 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.66@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:42:53.240341 30727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:53.234695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.66:44945)
0926 16:42:53.234954 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:53.234967 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:53.235060 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:53.235281 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:53.235283 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:53.235454 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:53.235549 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:53.235554 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:53.236270 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:53.236272 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:53.236824 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:53.236832 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:53.237353 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:53.237357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:53.237441 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:53.238076 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:53.238089 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:53.240222 (+ 2133us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":97,"threads_started":1}
W20260926 16:42:53.240470 30726 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:53.234774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:44945 (local address 127.21.12.126:38995)
0926 16:42:53.234902 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:53.234906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:53.235029 (+ 123us) server_negotiation.cc:407] Connection header received
0926 16:42:53.235133 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:53.235135 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:53.235176 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:53.235250 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:53.235648 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:53.236163 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:53.236967 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:53.237235 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:53.237351 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:53.240353 (+ 3002us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.66:44945: BlockingRecv error: recv got EOF from 127.21.12.66:44945 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 26 16:42:53 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.67@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:42:53 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.67@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:42:53.425143 30728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:53.419604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.67:46549)
0926 16:42:53.419943 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:53.419957 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:53.420053 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:53.420309 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:53.420311 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:53.420482 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:53.420578 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:53.420583 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:53.421382 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:53.421386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:53.421986 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:53.421996 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:53.422438 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:53.422441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:53.422506 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:53.423054 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:53.423066 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:53.425015 (+ 1949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":96,"threads_started":1}
W20260926 16:42:53.425280 30726 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:53.419735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:46549 (local address 127.21.12.126:38995)
0926 16:42:53.419896 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:53.419899 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:53.420031 (+ 132us) server_negotiation.cc:407] Connection header received
0926 16:42:53.420151 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:53.420153 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:53.420199 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:53.420268 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:53.420726 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:53.421259 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:53.422102 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:53.422326 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:53.422409 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:53.425162 (+ 2753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.67:46549: BlockingRecv error: recv got EOF from 127.21.12.67:46549 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 26 16:42:54 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.65@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:42:54 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.65@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:42:54.054145 30729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:54.048560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.65:35897)
0926 16:42:54.048835 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:54.048849 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:54.048944 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:54.049222 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:54.049224 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:54.049410 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:54.049507 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:54.049512 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:54.050398 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:54.050400 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:54.050963 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:54.050971 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:54.051452 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:54.051455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:54.051517 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:54.052103 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:54.052127 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:54.054032 (+ 1905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":76,"threads_started":1}
W20260926 16:42:54.054337 30730 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:54.048615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:35897 (local address 127.21.12.126:38995)
0926 16:42:54.048909 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:54.048913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:54.048927 (+ 14us) server_negotiation.cc:407] Connection header received
0926 16:42:54.049055 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:54.049058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:54.049103 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:54.049186 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:54.049651 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:54.050285 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:54.051100 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:54.051346 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:54.051437 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:54.054181 (+ 2744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.65:35897: BlockingRecv error: recv got EOF from 127.21.12.65:35897 (error 108)
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":90,"threads_started":1}
Sep 26 16:42:54 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.66@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:42:54 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.66@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:42:54.246608 30731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:54.241180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.66:52463)
0926 16:42:54.241438 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:54.241451 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:54.241545 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:54.241842 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:54.241844 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:54.242017 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:54.242113 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:54.242118 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:54.242908 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:54.242910 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:54.243487 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:54.243495 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:54.243927 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:54.243929 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:54.243995 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:54.244641 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:54.244670 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:54.246485 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":93,"threads_started":1}
W20260926 16:42:54.246726 30730 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:54.241248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:52463 (local address 127.21.12.126:38995)
0926 16:42:54.241385 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:54.241394 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:54.241502 (+ 108us) server_negotiation.cc:407] Connection header received
0926 16:42:54.241635 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:54.241637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:54.241681 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:54.241764 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:54.242242 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:54.242807 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:54.243618 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:54.243826 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:54.243902 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:54.246617 (+ 2715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.66:52463: BlockingRecv error: recv got EOF from 127.21.12.66:52463 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 16:42:54 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.67@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:42:54 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.67@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:42:54.431177 30732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:54.425986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.67:45773)
0926 16:42:54.426259 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:54.426273 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:54.426371 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:54.426596 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:54.426598 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:54.426771 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:54.426866 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:54.426872 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:54.427581 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:54.427584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:54.428141 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:54.428149 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:54.428557 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:54.428560 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:54.428621 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:54.429173 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:54.429184 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:54.431053 (+ 1869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":98,"threads_started":1}
W20260926 16:42:54.431331 30730 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:54.426065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:45773 (local address 127.21.12.126:38995)
0926 16:42:54.426187 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:54.426191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:54.426340 (+ 149us) server_negotiation.cc:407] Connection header received
0926 16:42:54.426448 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:54.426450 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:54.426496 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:54.426557 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:54.426964 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:54.427490 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:54.428247 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:54.428460 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:54.428537 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:54.431213 (+ 2676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.67:45773: BlockingRecv error: recv got EOF from 127.21.12.67:45773 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 26 16:42:55 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.65@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:42:55 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.65@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:42:55.064647 30733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:55.054967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.65:50849)
0926 16:42:55.055233 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:55.055246 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:55.055333 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:55.055574 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:55.055576 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:55.055755 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:55.055846 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:55.055851 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:55.056579 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:55.056581 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:55.057250 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:55.057260 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:55.057717 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:55.057719 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:55.057837 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:55.062688 (+ 4851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:55.062701 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:55.064530 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":88,"threads_started":1}
W20260926 16:42:55.064828 30734 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:55.055039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:50849 (local address 127.21.12.126:38995)
0926 16:42:55.055310 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:55.055314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:55.055328 (+ 14us) server_negotiation.cc:407] Connection header received
0926 16:42:55.055424 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:55.055426 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:55.055468 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:55.055543 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:55.055943 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:55.056484 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:55.057350 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:55.057597 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:55.057676 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:55.064658 (+ 6982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.65:50849: BlockingRecv error: recv got EOF from 127.21.12.65:50849 (error 108)
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":78,"threads_started":1}
Sep 26 16:42:55 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.66@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:42:55 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.66@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:42:55.253043 30735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:55.247393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.66:43265)
0926 16:42:55.247722 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:55.247735 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:55.247831 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:55.248045 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:55.248047 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:55.248233 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:55.248327 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:55.248332 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:55.249095 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:55.249098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:55.249662 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:55.249670 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:55.250280 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:55.250283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:55.250349 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:55.250927 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:55.250939 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:55.252877 (+ 1938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":99,"threads_started":1}
W20260926 16:42:55.253088 30734 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:55.247516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:43265 (local address 127.21.12.126:38995)
0926 16:42:55.247697 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:55.247701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:55.247805 (+ 104us) server_negotiation.cc:407] Connection header received
0926 16:42:55.247897 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:55.247899 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:55.247941 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:55.248008 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:55.248438 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:55.249000 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:55.249917 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:55.250175 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:55.250262 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:55.252990 (+ 2728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.66:43265: BlockingRecv error: recv got EOF from 127.21.12.66:43265 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 26 16:42:55 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.67@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:42:55 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.67@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:42:55.437389 30736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:55.432105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.67:39801)
0926 16:42:55.432394 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:55.432408 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:55.432510 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:55.432749 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:55.432751 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:55.432921 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:55.433012 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:55.433017 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:55.433716 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:55.433718 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:55.434435 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:55.434444 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:55.434882 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:55.434884 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:55.434944 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:55.435477 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:55.435488 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:55.437276 (+ 1788us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":103,"threads_started":1}
W20260926 16:42:55.437572 30734 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:55.432188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:39801 (local address 127.21.12.126:38995)
0926 16:42:55.432364 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:55.432367 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:55.432462 (+ 95us) server_negotiation.cc:407] Connection header received
0926 16:42:55.432577 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:55.432579 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:55.432623 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:55.432688 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:55.433110 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:55.433628 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:55.434545 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:55.434782 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:55.434863 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:55.437412 (+ 2549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.67:39801: BlockingRecv error: recv got EOF from 127.21.12.67:39801 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 26 16:42:56 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.65@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:42:56 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.65@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:42:56.071287 30737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:56.065416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.65:41293)
0926 16:42:56.065710 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:56.065724 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:56.065842 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:56.066068 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:56.066070 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:56.066248 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:56.066357 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:56.066362 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:56.067126 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:56.067128 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:56.067756 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:56.067767 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:56.068214 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:56.068217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:56.068364 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:56.068885 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:56.068898 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:56.071101 (+ 2203us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":94,"threads_started":1}
W20260926 16:42:56.071364 30738 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:56.065496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:41293 (local address 127.21.12.126:38995)
0926 16:42:56.065731 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:56.065734 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:56.065797 (+ 63us) server_negotiation.cc:407] Connection header received
0926 16:42:56.065907 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:56.065909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:56.065950 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:56.066024 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:56.066454 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:56.067002 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:56.067869 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:56.068088 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:56.068207 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:56.071239 (+ 3032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.65:41293: BlockingRecv error: recv got EOF from 127.21.12.65:41293 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":80,"threads_started":1}
Sep 26 16:42:56 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.66@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:42:56 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.66@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:42:56.259862 30739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:56.253963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.66:36855)
0926 16:42:56.254294 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:56.254310 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:56.254463 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:56.254754 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:56.254756 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:56.254973 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:56.255063 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:56.255070 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:56.255833 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:56.255838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:56.256417 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:56.256425 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:56.256887 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:56.256890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:56.256981 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:56.257532 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:56.257543 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:56.259749 (+ 2206us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":108,"threads_started":1}
W20260926 16:42:56.259989 30738 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:56.254145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:36855 (local address 127.21.12.126:38995)
0926 16:42:56.254295 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:56.254301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:56.254405 (+ 104us) server_negotiation.cc:407] Connection header received
0926 16:42:56.254574 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:56.254576 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:56.254643 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:56.254734 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:56.255189 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:56.255723 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:56.256535 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:56.256765 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:56.256886 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:56.259868 (+ 2982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.66:36855: BlockingRecv error: recv got EOF from 127.21.12.66:36855 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 16:42:56 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.67@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:42:56 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.67@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:42:56.444443 30740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:56.438291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.67:47809)
0926 16:42:56.438574 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:56.438593 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:56.438721 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:56.439101 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:56.439104 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:56.439301 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:56.439457 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:56.439465 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:56.440230 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:56.440234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:56.440824 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:56.440831 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:56.441291 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:56.441295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:56.441378 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:56.441946 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:56.441964 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:56.444285 (+ 2321us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":82,"threads_started":1}
W20260926 16:42:56.444638 30738 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:56.438348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:47809 (local address 127.21.12.126:38995)
0926 16:42:56.438493 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:56.438498 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:56.438670 (+ 172us) server_negotiation.cc:407] Connection header received
0926 16:42:56.438816 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:56.438839 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:56.438901 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:56.438977 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:56.439578 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:56.440110 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:56.440943 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:56.441180 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:56.441288 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:56.444517 (+ 3229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.67:47809: BlockingRecv error: recv got EOF from 127.21.12.67:47809 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 26 16:42:57 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.65@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:42:57 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.65@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:42:57.077903 30741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:57.072099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.65:56237)
0926 16:42:57.072406 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:57.072421 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:57.072513 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:57.072903 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:57.072905 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:57.073089 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:57.073188 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:57.073194 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:57.074076 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:57.074079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:57.074716 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:57.074726 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:57.075238 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:57.075241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:57.075309 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:57.075898 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:57.075910 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:57.077729 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":129,"threads_started":1}
W20260926 16:42:57.078095 30742 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:57.072212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:56237 (local address 127.21.12.126:38995)
0926 16:42:57.072462 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:57.072465 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:57.072478 (+ 13us) server_negotiation.cc:407] Connection header received
0926 16:42:57.072687 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:57.072690 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:57.072740 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:57.072829 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:57.073329 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:57.073959 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:57.074863 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:57.075087 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:57.075174 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:57.077907 (+ 2733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.65:56237: BlockingRecv error: recv got EOF from 127.21.12.65:56237 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":85,"threads_started":1}
Sep 26 16:42:57 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.66@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:42:57 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.66@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:42:57.266203 30743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:57.260709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.66:35583)
0926 16:42:57.261000 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:57.261014 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:57.261113 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:57.261346 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:57.261348 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:57.261525 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:57.261626 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:57.261632 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:57.262466 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:57.262469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:57.263165 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:57.263175 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:57.263683 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:57.263687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:57.263790 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:57.264335 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:57.264347 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:57.266094 (+ 1747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":96,"threads_started":1}
W20260926 16:42:57.266341 30742 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:57.260773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:35583 (local address 127.21.12.126:38995)
0926 16:42:57.260924 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:57.260929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:57.261084 (+ 155us) server_negotiation.cc:407] Connection header received
0926 16:42:57.261194 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:57.261196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:57.261243 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:57.261310 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:57.261723 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:57.262366 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:57.263248 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:57.263529 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:57.263640 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:57.266223 (+ 2583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.66:35583: BlockingRecv error: recv got EOF from 127.21.12.66:35583 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 16:42:57 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.67@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:42:57 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.67@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:42:57.451086 30744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:57.445309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.67:52213)
0926 16:42:57.445594 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:57.445608 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:57.445709 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:57.445998 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:57.446001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:57.446270 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:57.446390 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:57.446396 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:57.447150 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:57.447154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:57.447766 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:57.447774 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:57.448226 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:57.448230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:57.448316 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:57.448977 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:57.448990 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:57.450970 (+ 1980us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":104,"threads_started":1}
W20260926 16:42:57.451308 30742 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:57.445441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:52213 (local address 127.21.12.126:38995)
0926 16:42:57.445582 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:57.445585 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:57.445660 (+ 75us) server_negotiation.cc:407] Connection header received
0926 16:42:57.445842 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:57.445844 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:57.445887 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:57.445955 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:57.446504 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:57.447032 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:57.447873 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:57.448087 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:57.448164 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:57.451171 (+ 3007us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.67:52213: BlockingRecv error: recv got EOF from 127.21.12.67:52213 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 16:42:58 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.65@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:42:58 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.65@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:42:58.084501 30746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:58.078717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.65:56967)
0926 16:42:58.079030 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:58.079043 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:58.079137 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:58.079433 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:58.079436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:58.079630 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:58.079723 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:58.079730 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:58.080559 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:58.080561 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:58.081170 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:58.081179 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:58.081660 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:58.081662 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:58.081731 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:58.082513 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:58.082525 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:58.084331 (+ 1806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":97,"threads_started":1}
W20260926 16:42:58.084576 30745 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:58.078764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:56967 (local address 127.21.12.126:38995)
0926 16:42:58.079022 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:58.079025 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:58.079118 (+ 93us) server_negotiation.cc:407] Connection header received
0926 16:42:58.079220 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:58.079222 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:58.079268 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:58.079352 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:58.079853 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:58.080448 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:58.081299 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:58.081525 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:58.081632 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:58.084453 (+ 2821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.65:56967: BlockingRecv error: recv got EOF from 127.21.12.65:56967 (error 108)
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":98,"threads_started":1}
Sep 26 16:42:58 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.66@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:42:58 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.66@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:42:58.273307 30747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:58.267312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.66:32921)
0926 16:42:58.267655 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:58.267669 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:58.267798 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:58.268164 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:58.268168 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:58.268390 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:58.268487 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:58.268492 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:58.269330 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:58.269335 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:58.269943 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:58.269952 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:58.270469 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:58.270472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:58.270566 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:58.271114 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:58.271135 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:58.273194 (+ 2059us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":93,"threads_started":1}
W20260926 16:42:58.273499 30745 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:58.267306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:32921 (local address 127.21.12.126:38995)
0926 16:42:58.267477 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:58.267481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:58.267747 (+ 266us) server_negotiation.cc:407] Connection header received
0926 16:42:58.268010 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:58.268012 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:58.268053 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:58.268142 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:58.268603 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:58.269209 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:58.270070 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:58.270357 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:58.270467 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:58.273334 (+ 2867us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.66:32921: BlockingRecv error: recv got EOF from 127.21.12.66:32921 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 26 16:42:58 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.67@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:42:58 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.67@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:42:58.457690 30748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:58.451900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.67:46459)
0926 16:42:58.452249 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:58.452285 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:58.452407 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:58.452768 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:58.452771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:58.452970 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:58.453070 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:58.453076 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:58.453834 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:58.453837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:58.454386 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:58.454393 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:58.454929 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:58.454932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:58.454994 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:58.455462 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:58.455473 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:58.457514 (+ 2041us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":90,"threads_started":1}
W20260926 16:42:58.457715 30745 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:58.451942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:46459 (local address 127.21.12.126:38995)
0926 16:42:58.452086 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:58.452090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:58.452369 (+ 279us) server_negotiation.cc:407] Connection header received
0926 16:42:58.452514 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:58.452517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:58.452577 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:58.452730 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:58.453176 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:58.453735 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:58.454510 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:58.454813 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:58.454928 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:58.457625 (+ 2697us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.67:46459: BlockingRecv error: recv got EOF from 127.21.12.67:46459 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 16:42:59 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.65@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:42:59 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.65@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:42:59.090689 30749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:59.085370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.65:44577)
0926 16:42:59.085698 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:59.085713 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:59.085863 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:59.086091 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:59.086093 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:59.086284 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:59.086377 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:59.086383 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:59.087111 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:59.087113 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:59.087670 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:59.087679 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:59.088093 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:59.088095 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:59.088154 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:59.088643 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:59.088654 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:59.090577 (+ 1923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":101,"threads_started":1}
W20260926 16:42:59.090958 30750 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:59.085558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:44577 (local address 127.21.12.126:38995)
0926 16:42:59.085860 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:59.085863 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:59.085877 (+ 14us) server_negotiation.cc:407] Connection header received
0926 16:42:59.085934 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:59.085937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:59.085977 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:59.086052 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:59.086476 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:59.087022 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:59.087776 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:59.087998 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:59.088075 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:59.090798 (+ 2723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.65:44577: BlockingRecv error: recv got EOF from 127.21.12.65:44577 (error 108)
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":123,"threads_started":1}
Sep 26 16:42:59 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.66@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:42:59 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.66@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:42:59.279561 30751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:59.274052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.66:35625)
0926 16:42:59.274347 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:59.274359 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:59.274488 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:59.274785 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:59.274788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:59.274983 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:59.275074 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:59.275085 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:59.275998 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:59.276003 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:59.276561 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:59.276569 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:59.277031 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:59.277035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:59.277120 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:59.277664 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:59.277676 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:59.279451 (+ 1775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":80,"threads_started":1}
W20260926 16:42:59.279731 30750 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:59.274096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:35625 (local address 127.21.12.126:38995)
0926 16:42:59.274300 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:59.274304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:59.274428 (+ 124us) server_negotiation.cc:407] Connection header received
0926 16:42:59.274595 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:59.274598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:59.274644 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:59.274762 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:59.275214 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:59.275888 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:59.276677 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:59.276919 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:59.277029 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:59.279577 (+ 2548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.66:35625: BlockingRecv error: recv got EOF from 127.21.12.66:35625 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Sep 26 16:42:59 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.67@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:42:59 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.67@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:42:59.463857 30752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:59.458572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.67:46781)
0926 16:42:59.458839 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:59.458853 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:59.458939 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:59.459204 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:59.459207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:59.459383 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:59.459478 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:59.459483 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:59.460219 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:59.460222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:59.460779 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:59.460787 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:59.461191 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:59.461193 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:59.461256 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:59.461880 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:59.461892 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:59.463739 (+ 1847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":88,"threads_started":1}
W20260926 16:42:59.463992 30750 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:59.458778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:46781 (local address 127.21.12.126:38995)
0926 16:42:59.458893 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:59.458896 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:59.458908 (+ 12us) server_negotiation.cc:407] Connection header received
0926 16:42:59.459030 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:59.459032 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:59.459075 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:59.459141 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:59.459578 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:59.460121 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:59.460901 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:59.461104 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:59.461178 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:59.463874 (+ 2696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.67:46781: BlockingRecv error: recv got EOF from 127.21.12.67:46781 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 26 16:43:00 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.65@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:43:00 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.65@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:43:00.101420 30753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:00.091562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.65:43031)
0926 16:43:00.091866 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:00.091886 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:00.091997 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:00.092318 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:00.092321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:00.092544 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:00.092672 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:00.092680 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:00.093583 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:00.093585 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:00.094240 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:00.094251 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:00.094690 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:00.094692 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:00.094785 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:00.099116 (+ 4331us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:00.099129 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:00.101278 (+ 2149us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":87,"threads_started":1}
W20260926 16:43:00.101560 30754 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:43:00.091653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:43031 (local address 127.21.12.126:38995)
0926 16:43:00.091933 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:00.091937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:00.091972 (+ 35us) server_negotiation.cc:407] Connection header received
0926 16:43:00.092111 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:00.092115 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:00.092170 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:00.092281 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:00.092833 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:00.093479 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:00.094371 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:00.094589 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:00.094680 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:00.101423 (+ 6743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.65:43031: BlockingRecv error: recv got EOF from 127.21.12.65:43031 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":83,"threads_started":1}
Sep 26 16:43:00 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.66@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:43:00 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.66@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:43:00.285985 30755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:00.280230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.66:34509)
0926 16:43:00.280564 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:00.280621 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:00.280757 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:00.281021 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:00.281024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:00.281239 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:00.281329 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:00.281334 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:00.282162 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:00.282168 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:00.282735 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:00.282743 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:00.283194 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:00.283198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:00.283285 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:00.283776 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:00.283787 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:00.285872 (+ 2085us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":100,"threads_started":1}
W20260926 16:43:00.286224 30754 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:43:00.280349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:34509 (local address 127.21.12.126:38995)
0926 16:43:00.280522 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:00.280527 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:00.280703 (+ 176us) server_negotiation.cc:407] Connection header received
0926 16:43:00.280862 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:00.280864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:00.280917 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:00.281001 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:00.281449 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:00.282048 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:00.282855 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:00.283076 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:00.283192 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:00.286053 (+ 2861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.66:34509: BlockingRecv error: recv got EOF from 127.21.12.66:34509 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 26 16:43:00 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.67@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:43:00 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.67@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:43:00.470165 30756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:00.464758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.67:47377)
0926 16:43:00.465043 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:00.465057 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:00.465158 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:00.465386 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:00.465389 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:00.465564 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:00.465657 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:00.465662 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:00.466416 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:00.466418 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:00.466962 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:00.466969 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:00.467353 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:00.467356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:00.467413 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:00.467960 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:00.467972 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:00.470053 (+ 2081us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":104,"threads_started":1}
W20260926 16:43:00.470316 30754 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:43:00.464851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:47377 (local address 127.21.12.126:38995)
0926 16:43:00.465001 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:00.465006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:00.465110 (+ 104us) server_negotiation.cc:407] Connection header received
0926 16:43:00.465245 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:00.465247 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:00.465291 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:00.465348 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:00.465803 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:00.466324 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:00.467055 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:00.467263 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:00.467336 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:00.470179 (+ 2843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.67:47377: BlockingRecv error: recv got EOF from 127.21.12.67:47377 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 26 16:43:01 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.65@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:43:01 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.65@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:43:01.108059 30757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:01.102600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.65:45379)
0926 16:43:01.102853 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:01.102876 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:01.102999 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:01.103291 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:01.103297 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:01.103506 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:01.103601 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:01.103607 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:01.104396 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:01.104399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:01.104975 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:01.104984 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:01.105433 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:01.105435 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:01.105500 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:01.106093 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:01.106105 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:01.107947 (+ 1842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":89,"threads_started":1}
W20260926 16:43:01.108343 30758 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:43:01.102672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:45379 (local address 127.21.12.126:38995)
0926 16:43:01.102964 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:01.102969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:01.103011 (+ 42us) server_negotiation.cc:407] Connection header received
0926 16:43:01.103090 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:01.103094 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:01.103147 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:01.103258 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:01.103725 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:01.104293 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:01.105081 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:01.105328 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:01.105404 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:01.108086 (+ 2682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.65:45379: BlockingRecv error: recv got EOF from 127.21.12.65:45379 (error 108)
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":130,"threads_started":1}
Sep 26 16:43:01 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.66@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:43:01 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.66@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:43:01.292146 30759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:01.286738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.66:47429)
0926 16:43:01.287033 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:01.287055 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:01.287182 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:01.287497 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:01.287500 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:01.287699 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:01.287790 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:01.287795 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:01.288539 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:01.288544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:01.289103 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:01.289111 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:01.289575 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:01.289579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:01.289669 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:01.290228 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:01.290240 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:01.292032 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":82,"threads_started":1}
W20260926 16:43:01.292302 30758 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:43:01.286841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:47429 (local address 127.21.12.126:38995)
0926 16:43:01.286998 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:01.287003 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:01.287126 (+ 123us) server_negotiation.cc:407] Connection header received
0926 16:43:01.287332 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:01.287334 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:01.287380 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:01.287476 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:01.287910 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:01.288430 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:01.289221 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:01.289460 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:01.289572 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:01.292160 (+ 2588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.66:47429: BlockingRecv error: recv got EOF from 127.21.12.66:47429 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 16:43:01 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.67@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:43:01 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.67@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:43:01.476500 30760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:01.470930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.67:46123)
0926 16:43:01.471176 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:01.471189 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:01.471295 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:01.471526 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:01.471528 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:01.471700 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:01.471796 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:01.471801 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:01.472501 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:01.472503 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:01.473045 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:01.473052 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:01.473740 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:01.473744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:01.473857 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:01.474473 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:01.474484 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:01.476389 (+ 1905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":85,"threads_started":1}
W20260926 16:43:01.476637 30758 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:43:01.471077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:46123 (local address 127.21.12.126:38995)
0926 16:43:01.471201 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:01.471204 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:01.471237 (+ 33us) server_negotiation.cc:407] Connection header received
0926 16:43:01.471376 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:01.471379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:01.471425 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:01.471494 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:01.471891 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:01.472412 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:01.473326 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:01.473581 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:01.473700 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:01.476517 (+ 2817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.67:46123: BlockingRecv error: recv got EOF from 127.21.12.67:46123 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 26 16:43:02 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.65@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:43:02 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.65@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:43:02.114528 30761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:02.108884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.65:41991)
0926 16:43:02.109293 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:02.109307 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:02.109446 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:02.109848 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:02.109850 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:02.110025 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:02.110119 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:02.110124 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:02.110883 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:02.110886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:02.111444 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:02.111452 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:02.111918 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:02.111920 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:02.111979 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:02.112562 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:02.112574 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:02.114412 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":86,"threads_started":1}
W20260926 16:43:02.114765 30762 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:43:02.108977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:41991 (local address 127.21.12.126:38995)
0926 16:43:02.109262 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:02.109268 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:02.109376 (+ 108us) server_negotiation.cc:407] Connection header received
0926 16:43:02.109565 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:02.109571 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:02.109632 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:02.109817 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:02.110224 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:02.110779 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:02.111560 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:02.111822 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:02.111901 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:02.114626 (+ 2725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.65:41991: BlockingRecv error: recv got EOF from 127.21.12.65:41991 (error 108)
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":70,"threads_started":1}
Sep 26 16:43:02 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.66@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:43:02 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.66@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:43:02.299131 30763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:02.293111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.66:55457)
0926 16:43:02.293419 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:02.293433 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:02.293538 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:02.294006 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:02.294009 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:02.294218 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:02.294324 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:02.294330 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:02.295172 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:02.295174 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:02.295741 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:02.295750 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:02.296181 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:02.296183 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:02.296249 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:02.296763 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:02.296776 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:02.298957 (+ 2181us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":100,"threads_started":1}
W20260926 16:43:02.299269 30762 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:43:02.293201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:55457 (local address 127.21.12.126:38995)
0926 16:43:02.293356 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:02.293360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:02.293502 (+ 142us) server_negotiation.cc:407] Connection header received
0926 16:43:02.293620 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:02.293622 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:02.293707 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:02.293795 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:02.294477 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:02.295033 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:02.295858 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:02.296070 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:02.296150 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:02.299131 (+ 2981us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.66:55457: BlockingRecv error: recv got EOF from 127.21.12.66:55457 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 16:43:02 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.67@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:43:02 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.67@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:43:02.483085 30764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:02.477383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.67:44469)
0926 16:43:02.477704 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:02.477719 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:02.477879 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:02.478157 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:02.478160 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:02.478371 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:02.478459 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:02.478464 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:02.479228 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:02.479231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:02.479788 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:02.479795 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:02.480258 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:02.480261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:02.480345 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:02.480833 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:02.480845 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:02.482897 (+ 2052us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":82,"threads_started":1}
W20260926 16:43:02.483130 30762 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:43:02.477524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:44469 (local address 127.21.12.126:38995)
0926 16:43:02.477672 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:02.477676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:02.477813 (+ 137us) server_negotiation.cc:407] Connection header received
0926 16:43:02.477989 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:02.477993 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:02.478051 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:02.478136 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:02.478572 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:02.479106 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:02.479906 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:02.480144 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:02.480255 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:02.483022 (+ 2767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.67:44469: BlockingRecv error: recv got EOF from 127.21.12.67:44469 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 16:43:03 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.65@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:43:03 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.65@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:43:03.120877 30765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:03.115256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.65:55225)
0926 16:43:03.115544 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:03.115567 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:03.115660 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:03.115918 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:03.115920 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:03.116093 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:03.116198 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:03.116202 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:03.116970 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:03.116973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:03.117527 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:03.117534 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:03.117958 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:03.117960 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:03.118023 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:03.118674 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:03.118694 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:03.120697 (+ 2003us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":69,"threads_started":1}
W20260926 16:43:03.120946 30766 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:43:03.115391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:55225 (local address 127.21.12.126:38995)
0926 16:43:03.115637 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:03.115641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:03.115654 (+ 13us) server_negotiation.cc:407] Connection header received
0926 16:43:03.115747 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:03.115749 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:03.115792 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:03.115869 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:03.116336 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:03.116871 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:03.117628 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:03.117867 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:03.117941 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:03.120823 (+ 2882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.65:55225: BlockingRecv error: recv got EOF from 127.21.12.65:55225 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":78,"threads_started":1}
Sep 26 16:43:03 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.66@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:43:03 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.66@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:43:03.305951 30767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:03.299923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.66:35913)
0926 16:43:03.300169 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:03.300189 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:03.300289 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:03.300587 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:03.300593 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:03.300820 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:03.300947 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:03.300954 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:03.301870 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:03.301873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:03.302523 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:03.302532 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:03.303121 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:03.303128 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:03.303238 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:03.303747 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:03.303760 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:03.305720 (+ 1960us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":88,"threads_started":1}
W20260926 16:43:03.306198 30766 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:43:03.300002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:35913 (local address 127.21.12.126:38995)
0926 16:43:03.300134 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:03.300138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:03.300288 (+ 150us) server_negotiation.cc:407] Connection header received
0926 16:43:03.300367 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:03.300370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:03.300435 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:03.300549 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:03.301054 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:03.301695 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:03.302628 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:03.302995 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:03.303089 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:03.306043 (+ 2954us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.66:35913: BlockingRecv error: recv got EOF from 127.21.12.66:35913 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 26 16:43:03 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.67@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:43:03 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.67@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:43:03.489696 30768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:03.483917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.67:54753)
0926 16:43:03.484199 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:03.484213 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:03.484351 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:03.484626 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:03.484628 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:03.484909 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:03.485004 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:03.485011 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:03.485854 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:03.485858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:03.486417 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:03.486425 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:03.486892 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:03.486895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:03.486980 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:03.487471 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:03.487482 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:03.489521 (+ 2039us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":83,"threads_started":1}
W20260926 16:43:03.490044 30766 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:43:03.483981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:54753 (local address 127.21.12.126:38995)
0926 16:43:03.484174 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:03.484178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:03.484296 (+ 118us) server_negotiation.cc:407] Connection header received
0926 16:43:03.484461 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:03.484464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:03.484521 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:03.484606 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:03.485139 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:03.485697 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:03.486540 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:03.486780 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:03.486891 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:03.489819 (+ 2928us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.67:54753: BlockingRecv error: recv got EOF from 127.21.12.67:54753 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 26 16:43:04 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.65@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:43:04 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.65@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:43:04.127529 30769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:04.121743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.65:44397)
0926 16:43:04.122157 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:04.122170 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:04.122266 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:04.122540 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:04.122542 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:04.122717 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:04.122815 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:04.122821 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:04.123559 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:04.123561 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:04.124126 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:04.124133 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:04.124590 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:04.124593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:04.124662 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:04.125233 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:04.125251 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:04.127349 (+ 2098us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":222,"threads_started":1}
W20260926 16:43:04.127640 30770 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:43:04.121888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:44397 (local address 127.21.12.126:38995)
0926 16:43:04.122147 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:04.122150 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:04.122218 (+ 68us) server_negotiation.cc:407] Connection header received
0926 16:43:04.122362 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:04.122364 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:04.122411 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:04.122501 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:04.122925 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:04.123468 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:04.124222 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:04.124455 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:04.124530 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:04.127491 (+ 2961us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.65:44397: BlockingRecv error: recv got EOF from 127.21.12.65:44397 (error 108)
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":107,"threads_started":1}
Sep 26 16:43:04 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.66@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:43:04 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.66@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:43:04.313239 30771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:04.306888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.66:53763)
0926 16:43:04.307424 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:04.307440 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:04.307549 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:04.307812 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:04.307815 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:04.308043 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:04.308169 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:04.308178 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:04.309288 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:04.309293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:04.310079 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:04.310090 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:04.310576 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:04.310580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:04.310668 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:04.311170 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:04.311181 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:04.313107 (+ 1926us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":90,"threads_started":1}
W20260926 16:43:04.313366 30770 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:43:04.307329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:53763 (local address 127.21.12.126:38995)
0926 16:43:04.307486 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:04.307490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:04.307504 (+ 14us) server_negotiation.cc:407] Connection header received
0926 16:43:04.307638 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:04.307640 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:04.307684 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:04.307775 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:04.308290 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:04.309182 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:04.310206 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:04.310457 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:04.310573 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:04.313233 (+ 2660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.66:53763: BlockingRecv error: recv got EOF from 127.21.12.66:53763 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 26 16:43:04 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.67@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:43:04 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.67@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:43:04.495913 30772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:04.490477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.67:33751)
0926 16:43:04.490754 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:04.490768 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:04.490877 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:04.491111 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:04.491114 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:04.491307 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:04.491398 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:04.491403 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:04.492178 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:04.492180 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:04.492748 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:04.492757 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:04.493186 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:04.493189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:04.493258 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:04.493922 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:04.493935 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:04.495794 (+ 1859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":89,"threads_started":1}
W20260926 16:43:04.496152 30770 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:43:04.490548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:33751 (local address 127.21.12.126:38995)
0926 16:43:04.490711 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:04.490715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:04.490864 (+ 149us) server_negotiation.cc:407] Connection header received
0926 16:43:04.490945 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:04.490947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:04.490992 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:04.491075 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:04.491495 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:04.492055 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:04.492849 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:04.493056 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:04.493130 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:04.495998 (+ 2868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.67:33751: BlockingRecv error: recv got EOF from 127.21.12.67:33751 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 26 16:43:05 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.65@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:43:05 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.65@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:43:05.139408 30773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:05.128420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.65:57841)
0926 16:43:05.128787 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:05.128801 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:05.128896 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:05.129196 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:05.129199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:05.129397 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:05.129503 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:05.129509 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:05.130395 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:05.130398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:05.131026 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:05.131036 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:05.131561 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:05.131565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:05.131652 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:05.137153 (+ 5501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:05.137166 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:05.139209 (+ 2043us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":91,"threads_started":1}
W20260926 16:43:05.139489 30774 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:43:05.128572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:57841 (local address 127.21.12.126:38995)
0926 16:43:05.128943 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:05.128946 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:05.128960 (+ 14us) server_negotiation.cc:407] Connection header received
0926 16:43:05.129004 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:05.129006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:05.129049 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:05.129134 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:05.129628 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:05.130260 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:05.131164 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:05.131437 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:05.131555 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:05.139354 (+ 7799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.65:57841: BlockingRecv error: recv got EOF from 127.21.12.65:57841 (error 108)
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":142,"threads_started":1}
Sep 26 16:43:05 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.66@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:43:05 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.66@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:43:05.320472 30775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:05.314225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.66:49927)
0926 16:43:05.314586 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:05.314603 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:05.314733 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:05.315055 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:05.315058 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:05.315305 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:05.315412 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:05.315418 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:05.316368 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:05.316374 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:05.317007 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:05.317016 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:05.317622 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:05.317627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:05.317731 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:05.318359 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:05.318394 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:05.320290 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":133,"threads_started":1}
W20260926 16:43:05.320539 30774 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:43:05.314408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:49927 (local address 127.21.12.126:38995)
0926 16:43:05.314588 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:05.314594 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:05.314673 (+ 79us) server_negotiation.cc:407] Connection header received
0926 16:43:05.314865 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:05.314869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:05.314922 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:05.315027 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:05.315558 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:05.316239 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:05.317150 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:05.317488 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:05.317618 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:05.320423 (+ 2805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.66:49927: BlockingRecv error: recv got EOF from 127.21.12.66:49927 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 26 16:43:05 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.67@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:43:05 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.67@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:43:05.502604 30776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:05.496648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.67:32987)
0926 16:43:05.497119 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:05.497132 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:05.497283 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:05.497567 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:05.497570 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:05.497814 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:05.497905 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:05.497911 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:05.498665 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:05.498670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:05.499235 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:05.499242 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:05.499728 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:05.499732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:05.499818 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:05.500500 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:05.500512 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:05.502435 (+ 1923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":113,"threads_started":1}
W20260926 16:43:05.502681 30774 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:43:05.496758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:32987 (local address 127.21.12.126:38995)
0926 16:43:05.497031 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:05.497036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:05.497207 (+ 171us) server_negotiation.cc:407] Connection header received
0926 16:43:05.497406 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:05.497408 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:05.497454 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:05.497544 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:05.498024 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:05.498550 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:05.499371 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:05.499602 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:05.499713 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:05.502567 (+ 2854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.67:32987: BlockingRecv error: recv got EOF from 127.21.12.67:32987 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 16:43:06 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.65@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:43:06 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.65@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:43:06.145444 30777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:06.140347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.65:54047)
0926 16:43:06.140651 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:06.140665 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:06.140748 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:06.140979 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:06.140980 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:06.141158 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:06.141246 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:06.141251 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:06.141989 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:06.141992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:06.142555 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:06.142562 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:06.142973 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:06.142976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:06.143034 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:06.143518 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:06.143529 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:06.145332 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":106,"threads_started":1}
W20260926 16:43:06.145587 30778 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:43:06.140459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:54047 (local address 127.21.12.126:38995)
0926 16:43:06.140705 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:06.140709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:06.140721 (+ 12us) server_negotiation.cc:407] Connection header received
0926 16:43:06.140845 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:06.140847 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:06.140892 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:06.140967 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:06.141342 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:06.141899 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:06.142657 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:06.142882 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:06.142955 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:06.145456 (+ 2501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.65:54047: BlockingRecv error: recv got EOF from 127.21.12.65:54047 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":61,"threads_started":1}
Sep 26 16:43:06 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.66@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:43:06 dist-test-slave-n1hw krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.66@KRBTEST.COM for oryx/127.21.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:43:06.327142 30779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:06.321372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38995 (local address 127.21.12.66:60421)
0926 16:43:06.321652 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:06.321669 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:06.321815 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:06.322150 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:06.322153 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:06.322366 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:06.322459 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:06.322465 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:06.323544 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:06.323547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:06.324115 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:06.324124 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:06.324543 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:06.324546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:06.324609 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:06.325086 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:06.325097 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:06.327030 (+ 1933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.126:38995: Server oryx/127.21.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":77,"threads_started":1}
W20260926 16:43:06.327306 30778 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:43:06.321457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:60421 (local address 127.21.12.126:38995)
0926 16:43:06.321661 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:06.321664 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:06.321796 (+ 132us) server_negotiation.cc:407] Connection header received
0926 16:43:06.321950 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:06.321953 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:06.322007 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:06.322100 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:06.322700 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:06.323392 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:06.324223 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:06.324436 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:06.324511 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:06.327171 (+ 2660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.66:60421: BlockingRecv error: recv got EOF from 127.21.12.66:60421 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
I20260926 16:43:06.403425 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 30303
I20260926 16:43:06.408843 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 30455
I20260926 16:43:06.413959 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 30587
I20260926 16:43:06.419081 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 30233
2026-09-26T16:43:06Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17970 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T16:43:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:43:06Z Disabled control of system clock
I20260926 16:43:06.448863 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:39033
--webserver_interface=127.21.12.126
--webserver_port=0
--builtin_ntp_servers=127.21.12.84:45887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:39033
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 16:43:06.561121 30783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:06.561393 30783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:06.561429 30783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:06.565775 30783 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:43:06.565832 30783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:06.565847 30783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:06.565869 30783 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:43:06.565886 30783 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:43:06.570611 30783 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:45887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:39033
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:39033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:43:06.571653 30783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:06.573030 30783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:06.581837 30791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:06.582022 30783 server_base.cc:1034] running on GCE node
W20260926 16:43:06.581876 30788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:06.581851 30789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:06.582762 30783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:06.583921 30783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:06.585085 30783 hybrid_clock.cc:648] HybridClock initialized: now 1790440986585072 us; error 26 us; skew 500 ppm
I20260926 16:43:06.587148 30783 webserver.cc:460] Webserver started at http://127.21.12.126:43683/ using document root <none> and password file <none>
I20260926 16:43:06.587821 30783 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:06.587891 30783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:06.588156 30783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:06.590592 30783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "142935c0a68e4d37bfdc9b3f36c408eb"
format_stamp: "Formatted at 2026-09-26 16:43:06 on dist-test-slave-n1hw"
server_key: "84222003d3597142cb2db6a09d2a11cb"
server_key_iv: "d780c6f7184902b1ca82c24d89f29859"
server_key_version: "encryptionkey@0"
I20260926 16:43:06.591231 30783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "142935c0a68e4d37bfdc9b3f36c408eb"
format_stamp: "Formatted at 2026-09-26 16:43:06 on dist-test-slave-n1hw"
server_key: "84222003d3597142cb2db6a09d2a11cb"
server_key_iv: "d780c6f7184902b1ca82c24d89f29859"
server_key_version: "encryptionkey@0"
I20260926 16:43:06.595602 30783 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 16:43:06.598234 30797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:06.599505 30783 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:43:06.599663 30783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "142935c0a68e4d37bfdc9b3f36c408eb"
format_stamp: "Formatted at 2026-09-26 16:43:06 on dist-test-slave-n1hw"
server_key: "84222003d3597142cb2db6a09d2a11cb"
server_key_iv: "d780c6f7184902b1ca82c24d89f29859"
server_key_version: "encryptionkey@0"
I20260926 16:43:06.599834 30783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:06.618736 30783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:06.619434 30783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:06.619635 30783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:06.626447 30783 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.126:39033
I20260926 16:43:06.626482 30849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.126:39033 every 8 connection(s)
I20260926 16:43:06.627378 30783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:43:06.630652 30850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:06.634863 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 30783
I20260926 16:43:06.634989 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:43:06.635281 21553 external_mini_cluster.cc:1428] Setting key ae080a29f9735b68e1079c8ab7003be1
I20260926 16:43:06.636721 30850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 142935c0a68e4d37bfdc9b3f36c408eb: Bootstrap starting.
I20260926 16:43:06.639051 30850 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 142935c0a68e4d37bfdc9b3f36c408eb: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:06.639788 30850 log.cc:824] T 00000000000000000000000000000000 P 142935c0a68e4d37bfdc9b3f36c408eb: Log is configured to *not* fsync() on all Append() calls
I20260926 16:43:06.641911 30850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 142935c0a68e4d37bfdc9b3f36c408eb: No bootstrap required, opened a new log
I20260926 16:43:06.644567 30853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:06.636720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51904 (local address 127.21.12.126:39033)
0926 16:43:06.637375 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:06.637391 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:06.637426 (+ 35us) server_negotiation.cc:407] Connection header received
0926 16:43:06.638171 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:06.638188 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:06.638568 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:06.638847 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:06.639244 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:06.640286 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:06.641249 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:06.641742 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:06.641984 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:06.642857 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:06.642864 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:06.642880 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:43:06.643293 (+ 413us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:06.643356 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:43:06.643376 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:06.643489 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:06.643619 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 16:43:06.643885 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":138,"threads_started":1}
I20260926 16:43:06.646449 30850 raft_consensus.cc:348] T 00000000000000000000000000000000 P 142935c0a68e4d37bfdc9b3f36c408eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "142935c0a68e4d37bfdc9b3f36c408eb" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 39033 } }
I20260926 16:43:06.646708 30850 raft_consensus.cc:374] T 00000000000000000000000000000000 P 142935c0a68e4d37bfdc9b3f36c408eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:06.646803 30850 raft_consensus.cc:729] T 00000000000000000000000000000000 P 142935c0a68e4d37bfdc9b3f36c408eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 142935c0a68e4d37bfdc9b3f36c408eb, State: Initialized, Role: FOLLOWER
I20260926 16:43:06.647212 30850 consensus_queue.cc:260] T 00000000000000000000000000000000 P 142935c0a68e4d37bfdc9b3f36c408eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "142935c0a68e4d37bfdc9b3f36c408eb" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 39033 } }
I20260926 16:43:06.647331 30850 raft_consensus.cc:388] T 00000000000000000000000000000000 P 142935c0a68e4d37bfdc9b3f36c408eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:43:06.647377 30850 raft_consensus.cc:482] T 00000000000000000000000000000000 P 142935c0a68e4d37bfdc9b3f36c408eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:43:06.647449 30850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 142935c0a68e4d37bfdc9b3f36c408eb [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:06.648295 30850 raft_consensus.cc:504] T 00000000000000000000000000000000 P 142935c0a68e4d37bfdc9b3f36c408eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "142935c0a68e4d37bfdc9b3f36c408eb" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 39033 } }
I20260926 16:43:06.648572 30850 leader_election.cc:302] T 00000000000000000000000000000000 P 142935c0a68e4d37bfdc9b3f36c408eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 142935c0a68e4d37bfdc9b3f36c408eb; no voters:
I20260926 16:43:06.648834 30850 leader_election.cc:288] T 00000000000000000000000000000000 P 142935c0a68e4d37bfdc9b3f36c408eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:43:06.648962 30855 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 142935c0a68e4d37bfdc9b3f36c408eb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:43:06.649195 30855 raft_consensus.cc:686] T 00000000000000000000000000000000 P 142935c0a68e4d37bfdc9b3f36c408eb [term 1 LEADER]: Becoming Leader. State: Replica: 142935c0a68e4d37bfdc9b3f36c408eb, State: Running, Role: LEADER
I20260926 16:43:06.649556 30855 consensus_queue.cc:237] T 00000000000000000000000000000000 P 142935c0a68e4d37bfdc9b3f36c408eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "142935c0a68e4d37bfdc9b3f36c408eb" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 39033 } }
I20260926 16:43:06.650178 30850 sys_catalog.cc:564] T 00000000000000000000000000000000 P 142935c0a68e4d37bfdc9b3f36c408eb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:43:06.651417 30856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 142935c0a68e4d37bfdc9b3f36c408eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "142935c0a68e4d37bfdc9b3f36c408eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "142935c0a68e4d37bfdc9b3f36c408eb" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 39033 } } }
I20260926 16:43:06.651446 30857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 142935c0a68e4d37bfdc9b3f36c408eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 142935c0a68e4d37bfdc9b3f36c408eb. Latest consensus state: current_term: 1 leader_uuid: "142935c0a68e4d37bfdc9b3f36c408eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "142935c0a68e4d37bfdc9b3f36c408eb" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 39033 } } }
I20260926 16:43:06.651635 30856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 142935c0a68e4d37bfdc9b3f36c408eb [sys.catalog]: This master's current role is: LEADER
I20260926 16:43:06.651671 30857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 142935c0a68e4d37bfdc9b3f36c408eb [sys.catalog]: This master's current role is: LEADER
I20260926 16:43:06.652026 30864 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:43:06.655210 30864 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:43:06.660207 30864 catalog_manager.cc:1347] Generated new cluster ID: 444b6da09eed4aa9ad893fa6af41229c
I20260926 16:43:06.660267 30864 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:43:06.668074 30864 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:43:06.669067 30864 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:43:06.678725 30864 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 142935c0a68e4d37bfdc9b3f36c408eb: Generated new TSK 0
I20260926 16:43:06.679370 30864 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:43:06.690415 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.65:0
--local_ip_for_outbound_sockets=127.21.12.65
--webserver_interface=127.21.12.65
--webserver_port=0
--tserver_master_addrs=127.21.12.126:39033
--builtin_ntp_servers=127.21.12.84:45887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 16:43:06.799808 30874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:06.800048 30874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:06.800083 30874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:06.804303 30874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:06.804358 30874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:06.804409 30874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.65
I20260926 16:43:06.809128 30874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:45887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.65
--webserver_port=0
--tserver_master_addrs=127.21.12.126:39033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.65
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:43:06.810194 30874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:06.811554 30874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:06.821019 30882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:06.821082 30880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:06.821019 30879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:06.821298 30874 server_base.cc:1034] running on GCE node
I20260926 16:43:06.821887 30874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:06.822492 30874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:06.823683 30874 hybrid_clock.cc:648] HybridClock initialized: now 1790440986823668 us; error 32 us; skew 500 ppm
I20260926 16:43:06.830083 30874 webserver.cc:460] Webserver started at http://127.21.12.65:36755/ using document root <none> and password file <none>
I20260926 16:43:06.830693 30874 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:06.830832 30874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:06.831061 30874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:06.833114 30874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/data/instance:
uuid: "96fed03050044614bd6fe10744ae6240"
format_stamp: "Formatted at 2026-09-26 16:43:06 on dist-test-slave-n1hw"
server_key: "2d59c243b82cd4fcf52f65d1cfd9cdc7"
server_key_iv: "e6616c9b39d2a50ad284b19874735685"
server_key_version: "encryptionkey@0"
I20260926 16:43:06.833667 30874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance:
uuid: "96fed03050044614bd6fe10744ae6240"
format_stamp: "Formatted at 2026-09-26 16:43:06 on dist-test-slave-n1hw"
server_key: "2d59c243b82cd4fcf52f65d1cfd9cdc7"
server_key_iv: "e6616c9b39d2a50ad284b19874735685"
server_key_version: "encryptionkey@0"
I20260926 16:43:06.837359 30874 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:43:06.839947 30888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:06.841255 30874 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:43:06.841420 30874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/wal
uuid: "96fed03050044614bd6fe10744ae6240"
format_stamp: "Formatted at 2026-09-26 16:43:06 on dist-test-slave-n1hw"
server_key: "2d59c243b82cd4fcf52f65d1cfd9cdc7"
server_key_iv: "e6616c9b39d2a50ad284b19874735685"
server_key_version: "encryptionkey@0"
I20260926 16:43:06.841550 30874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:06.864249 30874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:06.864984 30874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:06.865185 30874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:06.865918 30874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:43:06.866884 30874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:43:06.866961 30874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:06.867026 30874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:43:06.867079 30874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:06.875780 30874 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.65:35731
I20260926 16:43:06.875805 31001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.65:35731 every 8 connection(s)
I20260926 16:43:06.876734 30874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
I20260926 16:43:06.886193 30853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:06.878608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:36445 (local address 127.21.12.126:39033)
0926 16:43:06.878754 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:06.878758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:06.879861 (+ 1103us) server_negotiation.cc:407] Connection header received
0926 16:43:06.881309 (+ 1448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:06.881313 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:06.881385 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:06.881491 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:06.882423 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:06.882960 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:06.884130 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:06.884390 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:06.884512 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:06.885656 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:06.885660 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:06.885663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:43:06.885683 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:06.885702 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:43:06.885711 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:06.885811 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:06.885986 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 16:43:06.886049 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 16:43:06.886852 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 30874
I20260926 16:43:06.886970 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance
I20260926 16:43:06.887275 21553 external_mini_cluster.cc:1428] Setting key 0773e8699206fed6df054ffbe5f3e7ed
I20260926 16:43:06.887302 31004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:06.878950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:39033 (local address 127.21.12.65:36445)
0926 16:43:06.879606 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:06.879646 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:06.881080 (+ 1434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:06.881671 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:06.881677 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:06.881778 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:06.882265 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:06.882276 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:06.883100 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:06.883104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:06.883993 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:06.884002 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:06.884557 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:06.884561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:06.884728 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:06.885383 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:43:06.885385 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:06.885428 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:43:06.885493 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:06.885824 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:06.885844 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:06.885853 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 16:43:06.885986 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 16:43:06.886279 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":105,"threads_started":1}
I20260926 16:43:06.888679 31002 heartbeater.cc:347] Connected to a master server at 127.21.12.126:39033
I20260926 16:43:06.888938 31002 heartbeater.cc:464] Registering TS with master...
I20260926 16:43:06.889598 31002 heartbeater.cc:511] Master 127.21.12.126:39033 requested a full tablet report, sending...
I20260926 16:43:06.889919 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.66:0
--local_ip_for_outbound_sockets=127.21.12.66
--webserver_interface=127.21.12.66
--webserver_port=0
--tserver_master_addrs=127.21.12.126:39033
--builtin_ntp_servers=127.21.12.84:45887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 16:43:06.891557 30813 ts_manager.cc:194] Registered new tserver with Master: 96fed03050044614bd6fe10744ae6240 (127.21.12.65:35731)
I20260926 16:43:06.892690 30813 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.12.65:36445
W20260926 16:43:07.012194 31005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:07.012485 31005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:07.012549 31005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:07.016667 31005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:07.016752 31005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:07.016836 31005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.66
I20260926 16:43:07.021641 31005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:45887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.66
--webserver_port=0
--tserver_master_addrs=127.21.12.126:39033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.66
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:43:07.022686 31005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:07.024024 31005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:07.033576 31010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:07.033596 31011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:07.033680 31005 server_base.cc:1034] running on GCE node
W20260926 16:43:07.033617 31013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:07.034289 31005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:07.034907 31005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:07.036067 31005 hybrid_clock.cc:648] HybridClock initialized: now 1790440987036033 us; error 44 us; skew 500 ppm
I20260926 16:43:07.037884 31005 webserver.cc:460] Webserver started at http://127.21.12.66:39405/ using document root <none> and password file <none>
I20260926 16:43:07.038443 31005 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:07.038504 31005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:07.038710 31005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:07.040691 31005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/data/instance:
uuid: "110626bd51ca4bb4afeaa813ceae85a9"
format_stamp: "Formatted at 2026-09-26 16:43:07 on dist-test-slave-n1hw"
server_key: "ad68cbdc1879f3c37c057ee267799f9e"
server_key_iv: "b402d8a425ac1ff99f26fa88d4f328f8"
server_key_version: "encryptionkey@0"
I20260926 16:43:07.041265 31005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance:
uuid: "110626bd51ca4bb4afeaa813ceae85a9"
format_stamp: "Formatted at 2026-09-26 16:43:07 on dist-test-slave-n1hw"
server_key: "ad68cbdc1879f3c37c057ee267799f9e"
server_key_iv: "b402d8a425ac1ff99f26fa88d4f328f8"
server_key_version: "encryptionkey@0"
I20260926 16:43:07.045024 31005 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 16:43:07.047770 31019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:07.049077 31005 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:43:07.049211 31005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/wal
uuid: "110626bd51ca4bb4afeaa813ceae85a9"
format_stamp: "Formatted at 2026-09-26 16:43:07 on dist-test-slave-n1hw"
server_key: "ad68cbdc1879f3c37c057ee267799f9e"
server_key_iv: "b402d8a425ac1ff99f26fa88d4f328f8"
server_key_version: "encryptionkey@0"
I20260926 16:43:07.049328 31005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:07.062095 31005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:07.062767 31005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:07.063007 31005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:07.063583 31005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:43:07.064507 31005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:43:07.064584 31005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:07.064651 31005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:43:07.064702 31005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:07.073580 31005 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.66:36137
I20260926 16:43:07.073606 31132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.66:36137 every 8 connection(s)
I20260926 16:43:07.074534 31005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
I20260926 16:43:07.076375 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 31005
I20260926 16:43:07.076511 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance
I20260926 16:43:07.076869 21553 external_mini_cluster.cc:1428] Setting key 8742e1f63253d9e9562f54c84d53b5b4
I20260926 16:43:07.079257 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.67:0
--local_ip_for_outbound_sockets=127.21.12.67
--webserver_interface=127.21.12.67
--webserver_port=0
--tserver_master_addrs=127.21.12.126:39033
--builtin_ntp_servers=127.21.12.84:45887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 16:43:07.085021 30853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:07.076417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:51739 (local address 127.21.12.126:39033)
0926 16:43:07.076570 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:07.076574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:07.077863 (+ 1289us) server_negotiation.cc:407] Connection header received
0926 16:43:07.079616 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:07.079619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:07.079661 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:07.079731 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:07.080938 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:07.081462 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:07.082726 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:07.082959 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:07.083079 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:07.084361 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:07.084368 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:07.084375 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:43:07.084393 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:07.084422 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:43:07.084436 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:07.084535 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:07.084724 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 16:43:07.084809 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 16:43:07.085745 31135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:07.076841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:39033 (local address 127.21.12.66:51739)
0926 16:43:07.077649 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:07.077704 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:07.079393 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:07.079920 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:07.079933 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:07.080049 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:07.080768 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:07.080792 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:07.081602 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:07.081606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:07.082582 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:07.082591 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:07.083118 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:07.083122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:07.083309 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:07.084039 (+ 730us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:43:07.084043 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:07.084097 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:43:07.084179 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:07.084562 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:07.084570 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:07.084576 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 16:43:07.084691 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 16:43:07.084941 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":149,"threads_started":1}
I20260926 16:43:07.087028 31133 heartbeater.cc:347] Connected to a master server at 127.21.12.126:39033
I20260926 16:43:07.087263 31133 heartbeater.cc:464] Registering TS with master...
I20260926 16:43:07.087765 31133 heartbeater.cc:511] Master 127.21.12.126:39033 requested a full tablet report, sending...
I20260926 16:43:07.088826 30813 ts_manager.cc:194] Registered new tserver with Master: 110626bd51ca4bb4afeaa813ceae85a9 (127.21.12.66:36137)
I20260926 16:43:07.089717 30813 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.12.66:51739
W20260926 16:43:07.191241 31136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:07.191499 31136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:07.191540 31136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:07.195535 31136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:07.195591 31136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:07.195653 31136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.67
I20260926 16:43:07.200312 31136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:45887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.67
--webserver_port=0
--tserver_master_addrs=127.21.12.126:39033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.67
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:43:07.201305 31136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:07.202726 31136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:07.212208 31142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:07.212241 31141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:07.212417 31136 server_base.cc:1034] running on GCE node
W20260926 16:43:07.212247 31144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:07.212870 31136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:07.213510 31136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:07.214694 31136 hybrid_clock.cc:648] HybridClock initialized: now 1790440987214631 us; error 72 us; skew 500 ppm
I20260926 16:43:07.216526 31136 webserver.cc:460] Webserver started at http://127.21.12.67:42003/ using document root <none> and password file <none>
I20260926 16:43:07.217092 31136 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:07.217186 31136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:07.217414 31136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:07.219491 31136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/data/instance:
uuid: "b6170f1cfae24b619ad1440b1e80dad0"
format_stamp: "Formatted at 2026-09-26 16:43:07 on dist-test-slave-n1hw"
server_key: "db3aa9e1241d105bde04e40c68b774bc"
server_key_iv: "f40a43508980cf58ad38f41f64470b7a"
server_key_version: "encryptionkey@0"
I20260926 16:43:07.220058 31136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance:
uuid: "b6170f1cfae24b619ad1440b1e80dad0"
format_stamp: "Formatted at 2026-09-26 16:43:07 on dist-test-slave-n1hw"
server_key: "db3aa9e1241d105bde04e40c68b774bc"
server_key_iv: "f40a43508980cf58ad38f41f64470b7a"
server_key_version: "encryptionkey@0"
I20260926 16:43:07.223949 31136 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 16:43:07.226730 31150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:07.228026 31136 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 16:43:07.228169 31136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/wal
uuid: "b6170f1cfae24b619ad1440b1e80dad0"
format_stamp: "Formatted at 2026-09-26 16:43:07 on dist-test-slave-n1hw"
server_key: "db3aa9e1241d105bde04e40c68b774bc"
server_key_iv: "f40a43508980cf58ad38f41f64470b7a"
server_key_version: "encryptionkey@0"
I20260926 16:43:07.228287 31136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:07.257679 31136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:07.258477 31136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:07.258683 31136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:07.259294 31136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:43:07.260429 31136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:43:07.260514 31136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:07.260586 31136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:43:07.260618 31136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:07.269309 31136 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.67:45485
I20260926 16:43:07.269387 31263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.67:45485 every 8 connection(s)
I20260926 16:43:07.270284 31136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
I20260926 16:43:07.276051 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 31136
I20260926 16:43:07.276142 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance
I20260926 16:43:07.276458 21553 external_mini_cluster.cc:1428] Setting key f11083cb0e373a71f42ece26429d5e96
I20260926 16:43:07.279852 30853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:07.272169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:52041 (local address 127.21.12.126:39033)
0926 16:43:07.272303 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:07.272307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:07.273382 (+ 1075us) server_negotiation.cc:407] Connection header received
0926 16:43:07.274851 (+ 1469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:07.274855 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:07.274915 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:07.275023 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:07.275936 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:07.276633 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:07.278081 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:07.278322 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:07.278446 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:07.279477 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:07.279480 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:07.279482 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:43:07.279494 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:07.279512 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:43:07.279520 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:07.279584 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:07.279693 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 16:43:07.279739 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 16:43:07.280663 31266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:07.272558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:39033 (local address 127.21.12.67:52041)
0926 16:43:07.273209 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:07.273250 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:07.274626 (+ 1376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:07.275194 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:07.275200 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:07.275287 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:07.275790 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:07.275803 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:07.276757 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:07.276761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:07.277964 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:07.277973 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:07.278478 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:07.278481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:07.278685 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:07.279220 (+ 535us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:43:07.279222 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:07.279259 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:43:07.279342 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:07.279585 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:07.279589 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:07.279592 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 16:43:07.279676 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 16:43:07.279892 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":105,"threads_started":1}
I20260926 16:43:07.281723 31264 heartbeater.cc:347] Connected to a master server at 127.21.12.126:39033
I20260926 16:43:07.281951 31264 heartbeater.cc:464] Registering TS with master...
I20260926 16:43:07.282397 31264 heartbeater.cc:511] Master 127.21.12.126:39033 requested a full tablet report, sending...
I20260926 16:43:07.283298 30813 ts_manager.cc:194] Registered new tserver with Master: b6170f1cfae24b619ad1440b1e80dad0 (127.21.12.67:45485)
I20260926 16:43:07.283852 30813 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.12.67:52041
I20260926 16:43:07.290318 21553 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 16:43:07.293042 30853 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:43:07.292250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51914 (local address 127.21.12.126:39033)
0926 16:43:07.292364 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:07.292368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:07.292547 (+ 179us) server_negotiation.cc:407] Connection header received
0926 16:43:07.292608 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:07.292610 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:07.292648 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:07.292708 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:07.292923 (+ 215us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51914: BlockingRecv error: recv got EOF from 127.0.0.1:51914 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 16:43:07.293519 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 30874
I20260926 16:43:07.298949 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 31005
I20260926 16:43:07.304191 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 31136
I20260926 16:43:07.309137 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 30783
2026-09-26T16:43:07Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (888 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T16:43:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:43:07Z Disabled control of system clock
I20260926 16:43:07.337493 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:37077
--webserver_interface=127.21.12.126
--webserver_port=0
--builtin_ntp_servers=127.21.12.84:38965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:37077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 16:43:07.443571 31275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:07.443845 31275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:07.443881 31275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:07.447732 31275 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:43:07.447781 31275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:07.447796 31275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:07.447813 31275 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:43:07.447826 31275 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:43:07.452373 31275 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:38965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:37077
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:37077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:43:07.453293 31275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:07.454562 31275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:07.462504 31280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:07.462502 31281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:07.462523 31283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:07.462872 31275 server_base.cc:1034] running on GCE node
I20260926 16:43:07.463376 31275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:07.464329 31275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:07.465483 31275 hybrid_clock.cc:648] HybridClock initialized: now 1790440987465460 us; error 42 us; skew 500 ppm
I20260926 16:43:07.467351 31275 webserver.cc:460] Webserver started at http://127.21.12.126:35805/ using document root <none> and password file <none>
I20260926 16:43:07.467893 31275 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:07.467953 31275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:07.468153 31275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:07.470172 31275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "a8b018b7d63c4b65a1c3bb9eea90d5fc"
format_stamp: "Formatted at 2026-09-26 16:43:07 on dist-test-slave-n1hw"
server_key: "cccbac4c4cab48b46ff0bb8e67626799"
server_key_iv: "748761f60f9b4a85d8c1fe0b01c615ac"
server_key_version: "encryptionkey@0"
I20260926 16:43:07.470714 31275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "a8b018b7d63c4b65a1c3bb9eea90d5fc"
format_stamp: "Formatted at 2026-09-26 16:43:07 on dist-test-slave-n1hw"
server_key: "cccbac4c4cab48b46ff0bb8e67626799"
server_key_iv: "748761f60f9b4a85d8c1fe0b01c615ac"
server_key_version: "encryptionkey@0"
I20260926 16:43:07.474077 31275 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:43:07.476239 31289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:07.477427 31275 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:43:07.477568 31275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "a8b018b7d63c4b65a1c3bb9eea90d5fc"
format_stamp: "Formatted at 2026-09-26 16:43:07 on dist-test-slave-n1hw"
server_key: "cccbac4c4cab48b46ff0bb8e67626799"
server_key_iv: "748761f60f9b4a85d8c1fe0b01c615ac"
server_key_version: "encryptionkey@0"
I20260926 16:43:07.477679 31275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:07.482008 31275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:07.482573 31275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:07.482756 31275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:07.489305 31275 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.126:37077
I20260926 16:43:07.489312 31341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.126:37077 every 8 connection(s)
I20260926 16:43:07.490264 31275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:43:07.492530 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 31275
I20260926 16:43:07.492645 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:43:07.493000 21553 external_mini_cluster.cc:1428] Setting key e6e186666681629e45da91a44d484db3
I20260926 16:43:07.493660 31342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:07.498478 31345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:07.494408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44872 (local address 127.21.12.126:37077)
0926 16:43:07.495008 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:07.495024 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:07.495059 (+ 35us) server_negotiation.cc:407] Connection header received
0926 16:43:07.495702 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:07.495718 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:07.496049 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:07.496328 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:07.497166 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:07.497171 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:07.497182 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:43:07.497192 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:07.497256 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:43:07.497272 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:07.497380 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:07.497510 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 16:43:07.497822 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":125,"threads_started":1}
I20260926 16:43:07.500152 31342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8b018b7d63c4b65a1c3bb9eea90d5fc: Bootstrap starting.
I20260926 16:43:07.502283 31342 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8b018b7d63c4b65a1c3bb9eea90d5fc: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:07.503023 31342 log.cc:824] T 00000000000000000000000000000000 P a8b018b7d63c4b65a1c3bb9eea90d5fc: Log is configured to *not* fsync() on all Append() calls
I20260926 16:43:07.504786 31342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8b018b7d63c4b65a1c3bb9eea90d5fc: No bootstrap required, opened a new log
I20260926 16:43:07.508140 31342 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8b018b7d63c4b65a1c3bb9eea90d5fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8b018b7d63c4b65a1c3bb9eea90d5fc" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 37077 } }
I20260926 16:43:07.508386 31342 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8b018b7d63c4b65a1c3bb9eea90d5fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:07.508477 31342 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8b018b7d63c4b65a1c3bb9eea90d5fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8b018b7d63c4b65a1c3bb9eea90d5fc, State: Initialized, Role: FOLLOWER
I20260926 16:43:07.508885 31342 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8b018b7d63c4b65a1c3bb9eea90d5fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8b018b7d63c4b65a1c3bb9eea90d5fc" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 37077 } }
I20260926 16:43:07.509035 31342 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8b018b7d63c4b65a1c3bb9eea90d5fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:43:07.509119 31342 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8b018b7d63c4b65a1c3bb9eea90d5fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:43:07.509245 31342 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8b018b7d63c4b65a1c3bb9eea90d5fc [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:07.510181 31342 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8b018b7d63c4b65a1c3bb9eea90d5fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8b018b7d63c4b65a1c3bb9eea90d5fc" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 37077 } }
I20260926 16:43:07.510511 31342 leader_election.cc:302] T 00000000000000000000000000000000 P a8b018b7d63c4b65a1c3bb9eea90d5fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8b018b7d63c4b65a1c3bb9eea90d5fc; no voters:
I20260926 16:43:07.510799 31342 leader_election.cc:288] T 00000000000000000000000000000000 P a8b018b7d63c4b65a1c3bb9eea90d5fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:43:07.510917 31347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8b018b7d63c4b65a1c3bb9eea90d5fc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:43:07.511178 31347 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8b018b7d63c4b65a1c3bb9eea90d5fc [term 1 LEADER]: Becoming Leader. State: Replica: a8b018b7d63c4b65a1c3bb9eea90d5fc, State: Running, Role: LEADER
I20260926 16:43:07.511507 31347 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8b018b7d63c4b65a1c3bb9eea90d5fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8b018b7d63c4b65a1c3bb9eea90d5fc" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 37077 } }
I20260926 16:43:07.511970 31342 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8b018b7d63c4b65a1c3bb9eea90d5fc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:43:07.513123 31349 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8b018b7d63c4b65a1c3bb9eea90d5fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8b018b7d63c4b65a1c3bb9eea90d5fc. Latest consensus state: current_term: 1 leader_uuid: "a8b018b7d63c4b65a1c3bb9eea90d5fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8b018b7d63c4b65a1c3bb9eea90d5fc" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 37077 } } }
I20260926 16:43:07.513163 31348 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8b018b7d63c4b65a1c3bb9eea90d5fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8b018b7d63c4b65a1c3bb9eea90d5fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8b018b7d63c4b65a1c3bb9eea90d5fc" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 37077 } } }
I20260926 16:43:07.513252 31349 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8b018b7d63c4b65a1c3bb9eea90d5fc [sys.catalog]: This master's current role is: LEADER
I20260926 16:43:07.513266 31348 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8b018b7d63c4b65a1c3bb9eea90d5fc [sys.catalog]: This master's current role is: LEADER
I20260926 16:43:07.513640 31356 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:43:07.516338 31356 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:43:07.521879 31356 catalog_manager.cc:1347] Generated new cluster ID: 118b75b621a34d83b0462ec36a139fd5
I20260926 16:43:07.521940 31356 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:43:07.531558 31356 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:43:07.531863 31356 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:43:07.535662 31356 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8b018b7d63c4b65a1c3bb9eea90d5fc: Generated new TSK 0
I20260926 16:43:07.536281 31356 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:43:07.545326 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.65:0
--local_ip_for_outbound_sockets=127.21.12.65
--webserver_interface=127.21.12.65
--webserver_port=0
--tserver_master_addrs=127.21.12.126:37077
--builtin_ntp_servers=127.21.12.84:38965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 16:43:07.655347 31366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:07.655583 31366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:07.655622 31366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:07.659623 31366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:07.659670 31366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:07.659729 31366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.65
I20260926 16:43:07.664346 31366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:38965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.65
--webserver_port=0
--tserver_master_addrs=127.21.12.126:37077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.65
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:43:07.665264 31366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:07.666531 31366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:07.676071 31374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:07.676168 31371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:07.676294 31366 server_base.cc:1034] running on GCE node
W20260926 16:43:07.676431 31372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:07.676873 31366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:07.677487 31366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:07.678666 31366 hybrid_clock.cc:648] HybridClock initialized: now 1790440987678624 us; error 60 us; skew 500 ppm
I20260926 16:43:07.680716 31366 webserver.cc:460] Webserver started at http://127.21.12.65:45957/ using document root <none> and password file <none>
I20260926 16:43:07.681304 31366 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:07.681409 31366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:07.681633 31366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:07.683671 31366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/data/instance:
uuid: "e8fc0afb56254ebd812dabd64d82f575"
format_stamp: "Formatted at 2026-09-26 16:43:07 on dist-test-slave-n1hw"
server_key: "c6ad2dd8fd40208c40f77454c3b5d25a"
server_key_iv: "cb3fac48efae948d67ce6656ada5c513"
server_key_version: "encryptionkey@0"
I20260926 16:43:07.684170 31366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance:
uuid: "e8fc0afb56254ebd812dabd64d82f575"
format_stamp: "Formatted at 2026-09-26 16:43:07 on dist-test-slave-n1hw"
server_key: "c6ad2dd8fd40208c40f77454c3b5d25a"
server_key_iv: "cb3fac48efae948d67ce6656ada5c513"
server_key_version: "encryptionkey@0"
I20260926 16:43:07.687641 31366 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 16:43:07.690097 31380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:07.691339 31366 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 16:43:07.691434 31366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/wal
uuid: "e8fc0afb56254ebd812dabd64d82f575"
format_stamp: "Formatted at 2026-09-26 16:43:07 on dist-test-slave-n1hw"
server_key: "c6ad2dd8fd40208c40f77454c3b5d25a"
server_key_iv: "cb3fac48efae948d67ce6656ada5c513"
server_key_version: "encryptionkey@0"
I20260926 16:43:07.691521 31366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:07.695770 31366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:07.696296 31366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:07.696427 31366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:07.696887 31366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:43:07.697670 31366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:43:07.697715 31366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:07.697814 31366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:43:07.697847 31366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:07.706637 31366 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.65:33697
I20260926 16:43:07.706662 31493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.65:33697 every 8 connection(s)
I20260926 16:43:07.707496 31366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
I20260926 16:43:07.711207 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 31366
I20260926 16:43:07.711359 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance
I20260926 16:43:07.711653 21553 external_mini_cluster.cc:1428] Setting key ec8707f2d76a0aa66add5e7ee99ff870
I20260926 16:43:07.714003 31345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:07.709425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:39419 (local address 127.21.12.126:37077)
0926 16:43:07.709574 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:07.709577 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:07.710633 (+ 1056us) server_negotiation.cc:407] Connection header received
0926 16:43:07.712028 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:07.712031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:07.712075 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:07.712164 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:07.713485 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:07.713489 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:07.713492 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:43:07.713503 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:07.713525 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:43:07.713535 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:07.713627 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:07.713834 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 16:43:07.713877 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 16:43:07.713999 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.66:0
--local_ip_for_outbound_sockets=127.21.12.66
--webserver_interface=127.21.12.66
--webserver_port=0
--tserver_master_addrs=127.21.12.126:37077
--builtin_ntp_servers=127.21.12.84:38965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 16:43:07.714890 31496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:07.709852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:37077 (local address 127.21.12.65:39419)
0926 16:43:07.710488 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:07.710527 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:07.711818 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:07.712372 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:07.712381 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:07.712489 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:07.713178 (+ 689us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:43:07.713181 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:07.713249 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:43:07.713337 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:07.713624 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:07.713631 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:07.713635 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 16:43:07.713826 (+ 191us) client_negotiation.cc:240] Negotiation successful
0926 16:43:07.714055 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":111,"threads_started":1}
I20260926 16:43:07.716120 31494 heartbeater.cc:347] Connected to a master server at 127.21.12.126:37077
I20260926 16:43:07.716346 31494 heartbeater.cc:464] Registering TS with master...
I20260926 16:43:07.716734 31494 heartbeater.cc:511] Master 127.21.12.126:37077 requested a full tablet report, sending...
I20260926 16:43:07.718250 31306 ts_manager.cc:194] Registered new tserver with Master: e8fc0afb56254ebd812dabd64d82f575 (127.21.12.65:33697)
W20260926 16:43:07.822441 31497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:07.822665 31497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:07.822738 31497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:07.826628 31497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:07.826673 31497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:07.826781 31497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.66
I20260926 16:43:07.831439 31497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:38965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.66
--webserver_port=0
--tserver_master_addrs=127.21.12.126:37077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31497
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.66
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:43:07.832383 31497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:07.833683 31497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:07.842892 31502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:07.842909 31503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:07.842913 31505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:07.843283 31497 server_base.cc:1034] running on GCE node
I20260926 16:43:07.843618 31497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:07.844179 31497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:07.845382 31497 hybrid_clock.cc:648] HybridClock initialized: now 1790440987845372 us; error 26 us; skew 500 ppm
I20260926 16:43:07.847178 31497 webserver.cc:460] Webserver started at http://127.21.12.66:34451/ using document root <none> and password file <none>
I20260926 16:43:07.847754 31497 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:07.847847 31497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:07.848063 31497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:07.850081 31497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/data/instance:
uuid: "b070b558777a47e08761b7455b0165f0"
format_stamp: "Formatted at 2026-09-26 16:43:07 on dist-test-slave-n1hw"
server_key: "a7782ca7d86c85eb13131739275f7df5"
server_key_iv: "4a240a96bf220eae73e328814fcb99e0"
server_key_version: "encryptionkey@0"
I20260926 16:43:07.850626 31497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance:
uuid: "b070b558777a47e08761b7455b0165f0"
format_stamp: "Formatted at 2026-09-26 16:43:07 on dist-test-slave-n1hw"
server_key: "a7782ca7d86c85eb13131739275f7df5"
server_key_iv: "4a240a96bf220eae73e328814fcb99e0"
server_key_version: "encryptionkey@0"
I20260926 16:43:07.854251 31497 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:43:07.856602 31511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:07.857730 31497 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:43:07.857918 31497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/wal
uuid: "b070b558777a47e08761b7455b0165f0"
format_stamp: "Formatted at 2026-09-26 16:43:07 on dist-test-slave-n1hw"
server_key: "a7782ca7d86c85eb13131739275f7df5"
server_key_iv: "4a240a96bf220eae73e328814fcb99e0"
server_key_version: "encryptionkey@0"
I20260926 16:43:07.858028 31497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:07.862376 31497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:07.862946 31497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:07.863126 31497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:07.863641 31497 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:43:07.864495 31497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:43:07.864568 31497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:07.864631 31497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:43:07.864665 31497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:07.873481 31497 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.66:42487
I20260926 16:43:07.873510 31624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.66:42487 every 8 connection(s)
I20260926 16:43:07.874449 31497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
I20260926 16:43:07.879863 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 31497
I20260926 16:43:07.879992 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance
I20260926 16:43:07.880352 21553 external_mini_cluster.cc:1428] Setting key 8d52068df246afc139393d130d7557df
I20260926 16:43:07.880700 31345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:07.876288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:33425 (local address 127.21.12.126:37077)
0926 16:43:07.876451 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:07.876455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:07.877427 (+ 972us) server_negotiation.cc:407] Connection header received
0926 16:43:07.878742 (+ 1315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:07.878745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:07.878782 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:07.878866 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:07.880148 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:07.880153 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:07.880157 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:43:07.880175 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:07.880201 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:43:07.880214 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:07.880324 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:07.880523 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 16:43:07.880570 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 16:43:07.881687 31627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:07.876632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:37077 (local address 127.21.12.66:33425)
0926 16:43:07.877251 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:07.877305 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:07.878557 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:07.879026 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:07.879032 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:07.879118 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:07.879769 (+ 651us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:43:07.879770 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:07.879810 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:43:07.879882 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:07.880310 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:07.880316 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:07.880320 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 16:43:07.880508 (+ 188us) client_negotiation.cc:240] Negotiation successful
0926 16:43:07.880723 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":104,"threads_started":1}
I20260926 16:43:07.882503 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.67:0
--local_ip_for_outbound_sockets=127.21.12.67
--webserver_interface=127.21.12.67
--webserver_port=0
--tserver_master_addrs=127.21.12.126:37077
--builtin_ntp_servers=127.21.12.84:38965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 16:43:07.882956 31625 heartbeater.cc:347] Connected to a master server at 127.21.12.126:37077
I20260926 16:43:07.883189 31625 heartbeater.cc:464] Registering TS with master...
I20260926 16:43:07.883606 31625 heartbeater.cc:511] Master 127.21.12.126:37077 requested a full tablet report, sending...
I20260926 16:43:07.884688 31306 ts_manager.cc:194] Registered new tserver with Master: b070b558777a47e08761b7455b0165f0 (127.21.12.66:42487)
W20260926 16:43:07.995112 31628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:07.995419 31628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:07.995460 31628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:07.999541 31628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:07.999591 31628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:07.999670 31628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.67
I20260926 16:43:08.004451 31628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:38965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.67
--webserver_port=0
--tserver_master_addrs=127.21.12.126:37077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.67
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:43:08.005446 31628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:08.006816 31628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 16:43:08.016209 31628 server_base.cc:1034] running on GCE node
W20260926 16:43:08.016213 31634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:08.016345 31633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:08.016243 31636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:08.016808 31628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:08.017405 31628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:08.018738 31628 hybrid_clock.cc:648] HybridClock initialized: now 1790440988018687 us; error 61 us; skew 500 ppm
I20260926 16:43:08.020532 31628 webserver.cc:460] Webserver started at http://127.21.12.67:36269/ using document root <none> and password file <none>
I20260926 16:43:08.021111 31628 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:08.021173 31628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:08.021421 31628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:08.023473 31628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/data/instance:
uuid: "05df82451255460f9d09a12871ff92a7"
format_stamp: "Formatted at 2026-09-26 16:43:08 on dist-test-slave-n1hw"
server_key: "4a151e47722b58e8840923943cec8d9c"
server_key_iv: "56fa63b3bffb8d564ac34fed11f1884c"
server_key_version: "encryptionkey@0"
I20260926 16:43:08.024027 31628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance:
uuid: "05df82451255460f9d09a12871ff92a7"
format_stamp: "Formatted at 2026-09-26 16:43:08 on dist-test-slave-n1hw"
server_key: "4a151e47722b58e8840923943cec8d9c"
server_key_iv: "56fa63b3bffb8d564ac34fed11f1884c"
server_key_version: "encryptionkey@0"
I20260926 16:43:08.027580 31628 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 16:43:08.030092 31642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:08.031296 31628 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 16:43:08.031414 31628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/wal
uuid: "05df82451255460f9d09a12871ff92a7"
format_stamp: "Formatted at 2026-09-26 16:43:08 on dist-test-slave-n1hw"
server_key: "4a151e47722b58e8840923943cec8d9c"
server_key_iv: "56fa63b3bffb8d564ac34fed11f1884c"
server_key_version: "encryptionkey@0"
I20260926 16:43:08.031512 31628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:08.035844 31628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:08.036433 31628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:08.036607 31628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:08.037132 31628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:43:08.038043 31628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:43:08.038090 31628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:08.038156 31628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:43:08.038192 31628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:08.046478 31628 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.67:46265
I20260926 16:43:08.046496 31755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.67:46265 every 8 connection(s)
I20260926 16:43:08.047385 31628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
I20260926 16:43:08.049266 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 31628
I20260926 16:43:08.049381 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance
I20260926 16:43:08.049726 21553 external_mini_cluster.cc:1428] Setting key 603f346d580172c2ae2309be16c6a7b6
I20260926 16:43:08.053457 31345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:08.049178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:44669 (local address 127.21.12.126:37077)
0926 16:43:08.049312 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:08.049316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:08.050770 (+ 1454us) server_negotiation.cc:407] Connection header received
0926 16:43:08.051995 (+ 1225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:08.051998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:08.052031 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:08.052095 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:08.053081 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:08.053084 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:08.053087 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:43:08.053097 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:08.053117 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:43:08.053124 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:08.053192 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:08.053316 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 16:43:08.053352 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 16:43:08.054342 31758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:08.049822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:37077 (local address 127.21.12.67:44669)
0926 16:43:08.050619 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:08.050668 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:08.051821 (+ 1153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:08.052256 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:08.052263 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:08.052362 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:08.052852 (+ 490us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:43:08.052853 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:08.052892 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:43:08.052956 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:08.053199 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:08.053205 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:08.053209 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 16:43:08.053346 (+ 137us) client_negotiation.cc:240] Negotiation successful
0926 16:43:08.053528 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":163,"threads_started":1}
I20260926 16:43:08.055421 31756 heartbeater.cc:347] Connected to a master server at 127.21.12.126:37077
I20260926 16:43:08.055604 31756 heartbeater.cc:464] Registering TS with master...
I20260926 16:43:08.055980 31756 heartbeater.cc:511] Master 127.21.12.126:37077 requested a full tablet report, sending...
I20260926 16:43:08.056864 31306 ts_manager.cc:194] Registered new tserver with Master: 05df82451255460f9d09a12871ff92a7 (127.21.12.67:46265)
I20260926 16:43:08.063402 21553 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 16:43:08.066228 31345 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:43:08.065287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44886 (local address 127.21.12.126:37077)
0926 16:43:08.065427 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:08.065431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:08.065619 (+ 188us) server_negotiation.cc:407] Connection header received
0926 16:43:08.065698 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:08.065700 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:08.065730 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:08.065809 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:08.066104 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44886: BlockingRecv error: recv got EOF from 127.0.0.1:44886 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 16:43:08.066354 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 31366
I20260926 16:43:08.072183 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 31497
I20260926 16:43:08.077421 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 31628
I20260926 16:43:08.082692 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 31275
2026-09-26T16:43:08Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (773 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:43:08 dist-test-slave-n1hw krb5kdc[31766](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:43:08 dist-test-slave-n1hw krb5kdc[31766](info): set up 2 sockets
Sep 26 16:43:08 dist-test-slave-n1hw krb5kdc[31766](info): commencing operation
krb5kdc: starting...
W20260926 16:43:09.132880 21553 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Sep 26 16:43:09 dist-test-slave-n1hw krb5kdc[31766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:43:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:43:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
I20260926 16:43:09.256227 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:35049
--webserver_interface=127.21.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:46439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:35049
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:43:09.364611 31779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:09.364925 31779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:09.365008 31779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:09.369059 31779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:43:09.369156 31779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:09.369191 31779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:09.369226 31779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:43:09.369254 31779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:43:09.374470 31779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:46439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:35049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:35049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:43:09.375500 31779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:09.376521 31779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:09.385098 31787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:09.385106 31784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:09.385106 31785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:09.385385 31779 server_base.cc:1034] running on GCE node
I20260926 16:43:09.386071 31779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:09.387127 31779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:09.388288 31779 hybrid_clock.cc:648] HybridClock initialized: now 1790440989388273 us; error 27 us; skew 500 ppm
Sep 26 16:43:09 dist-test-slave-n1hw krb5kdc[31766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440989, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:43:09.391057 31779 init.cc:377] Logged in from keytab as kudu/127.21.12.126@KRBTEST.COM (short username kudu)
I20260926 16:43:09.391956 31779 webserver.cc:460] Webserver started at http://127.21.12.126:38097/ using document root <none> and password file <none>
I20260926 16:43:09.392498 31779 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:09.392556 31779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:09.392778 31779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:09.394829 31779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "f5319cda62874eb789beb86cc8248b18"
format_stamp: "Formatted at 2026-09-26 16:43:09 on dist-test-slave-n1hw"
server_key: "f57ba9968e9d9a5daf95e3c844cabe3e"
server_key_iv: "1c6b25e94e47ee38a4c151659d77c9fb"
server_key_version: "encryptionkey@0"
I20260926 16:43:09.395394 31779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "f5319cda62874eb789beb86cc8248b18"
format_stamp: "Formatted at 2026-09-26 16:43:09 on dist-test-slave-n1hw"
server_key: "f57ba9968e9d9a5daf95e3c844cabe3e"
server_key_iv: "1c6b25e94e47ee38a4c151659d77c9fb"
server_key_version: "encryptionkey@0"
I20260926 16:43:09.398975 31779 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:43:09.401268 31794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:09.402631 31779 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:43:09.402772 31779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "f5319cda62874eb789beb86cc8248b18"
format_stamp: "Formatted at 2026-09-26 16:43:09 on dist-test-slave-n1hw"
server_key: "f57ba9968e9d9a5daf95e3c844cabe3e"
server_key_iv: "1c6b25e94e47ee38a4c151659d77c9fb"
server_key_version: "encryptionkey@0"
I20260926 16:43:09.402889 31779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:09.434952 31779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:09.437960 31779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:09.438179 31779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:09.445133 31779 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.126:35049
I20260926 16:43:09.445420 31846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.126:35049 every 8 connection(s)
I20260926 16:43:09.446128 31779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:43:09.448838 31847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:09.452454 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 31779
I20260926 16:43:09.452596 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:43:09.452867 21553 external_mini_cluster.cc:1428] Setting key df5183bca4b7b07785bfc9e26ee09414
I20260926 16:43:09.454591 31847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5319cda62874eb789beb86cc8248b18: Bootstrap starting.
I20260926 16:43:09.457068 31847 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5319cda62874eb789beb86cc8248b18: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:09.457851 31847 log.cc:824] T 00000000000000000000000000000000 P f5319cda62874eb789beb86cc8248b18: Log is configured to *not* fsync() on all Append() calls
I20260926 16:43:09.459941 31847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5319cda62874eb789beb86cc8248b18: No bootstrap required, opened a new log
Sep 26 16:43:09 dist-test-slave-n1hw krb5kdc[31766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:43:09.463498 31847 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5319cda62874eb789beb86cc8248b18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5319cda62874eb789beb86cc8248b18" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 35049 } }
I20260926 16:43:09.463798 31847 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5319cda62874eb789beb86cc8248b18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:09.463891 31847 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5319cda62874eb789beb86cc8248b18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5319cda62874eb789beb86cc8248b18, State: Initialized, Role: FOLLOWER
I20260926 16:43:09.464311 31847 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5319cda62874eb789beb86cc8248b18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5319cda62874eb789beb86cc8248b18" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 35049 } }
I20260926 16:43:09.464462 31847 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5319cda62874eb789beb86cc8248b18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:43:09.464547 31847 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5319cda62874eb789beb86cc8248b18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:43:09.464643 31847 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5319cda62874eb789beb86cc8248b18 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:09.465443 31847 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5319cda62874eb789beb86cc8248b18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5319cda62874eb789beb86cc8248b18" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 35049 } }
I20260926 16:43:09.465814 31847 leader_election.cc:302] T 00000000000000000000000000000000 P f5319cda62874eb789beb86cc8248b18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5319cda62874eb789beb86cc8248b18; no voters:
I20260926 16:43:09.466118 31847 leader_election.cc:288] T 00000000000000000000000000000000 P f5319cda62874eb789beb86cc8248b18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:43:09.466459 31853 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5319cda62874eb789beb86cc8248b18 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:43:09.466820 31853 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5319cda62874eb789beb86cc8248b18 [term 1 LEADER]: Becoming Leader. State: Replica: f5319cda62874eb789beb86cc8248b18, State: Running, Role: LEADER
I20260926 16:43:09.467149 31853 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5319cda62874eb789beb86cc8248b18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5319cda62874eb789beb86cc8248b18" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 35049 } }
I20260926 16:43:09.467396 31847 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5319cda62874eb789beb86cc8248b18 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:43:09.467821 31850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:09.454284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60848 (local address 127.21.12.126:35049)
0926 16:43:09.454671 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:09.454691 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:09.454752 (+ 61us) server_negotiation.cc:407] Connection header received
0926 16:43:09.455359 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:09.455376 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:09.455683 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:09.456134 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:09.457137 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:09.458389 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:09.459531 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:09.459979 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:09.460148 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:09.462652 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:09.462658 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:09.462674 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:09.462706 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:09.464664 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:09.465079 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:09.465084 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:09.465090 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:09.465207 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:09.465496 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:09.465501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:09.465502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:09.466037 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:09.466259 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:09.466556 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 16:43:09.466861 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":130,"threads_started":1}
I20260926 16:43:09.468823 31854 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5319cda62874eb789beb86cc8248b18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5319cda62874eb789beb86cc8248b18. Latest consensus state: current_term: 1 leader_uuid: "f5319cda62874eb789beb86cc8248b18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5319cda62874eb789beb86cc8248b18" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 35049 } } }
I20260926 16:43:09.468948 31854 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5319cda62874eb789beb86cc8248b18 [sys.catalog]: This master's current role is: LEADER
I20260926 16:43:09.469290 31852 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5319cda62874eb789beb86cc8248b18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5319cda62874eb789beb86cc8248b18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5319cda62874eb789beb86cc8248b18" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 35049 } } }
I20260926 16:43:09.469401 31852 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5319cda62874eb789beb86cc8248b18 [sys.catalog]: This master's current role is: LEADER
I20260926 16:43:09.469348 31863 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:43:09.472378 31863 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:43:09.477941 31863 catalog_manager.cc:1347] Generated new cluster ID: 77c109d33ece4a33afbce85e57eb0457
I20260926 16:43:09.478003 31863 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:43:09.493999 31863 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:43:09.494968 31863 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:43:09.502506 31863 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5319cda62874eb789beb86cc8248b18: Generated new TSK 0
I20260926 16:43:09.503298 31863 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab.
Entry for principal kudu/127.21.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab.
I20260926 16:43:09.545050 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.65:0
--local_ip_for_outbound_sockets=127.21.12.65
--webserver_interface=127.21.12.65
--webserver_port=0
--tserver_master_addrs=127.21.12.126:35049
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab
--principal=kudu/127.21.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:46439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:43:09.665282 31873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:09.665601 31873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:09.665675 31873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:09.669979 31873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:09.670051 31873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:09.670135 31873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.65
I20260926 16:43:09.675276 31873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:46439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab
--principal=kudu/127.21.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.65
--webserver_port=0
--tserver_master_addrs=127.21.12.126:35049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.65
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:43:09.676244 31873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:09.677224 31873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:09.686538 31878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:09.686514 31881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:09.686522 31879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:09.686672 31873 server_base.cc:1034] running on GCE node
I20260926 16:43:09.687125 31873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:09.687721 31873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:09.688894 31873 hybrid_clock.cc:648] HybridClock initialized: now 1790440989688880 us; error 25 us; skew 500 ppm
Sep 26 16:43:09 dist-test-slave-n1hw krb5kdc[31766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440989, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:43:09.691612 31873 init.cc:377] Logged in from keytab as kudu/127.21.12.65@KRBTEST.COM (short username kudu)
I20260926 16:43:09.692387 31873 webserver.cc:460] Webserver started at http://127.21.12.65:38093/ using document root <none> and password file <none>
I20260926 16:43:09.692929 31873 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:09.693019 31873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:09.693233 31873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:09.695165 31873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/data/instance:
uuid: "677235c3ea774fe8b0a5a2b56c34fde0"
format_stamp: "Formatted at 2026-09-26 16:43:09 on dist-test-slave-n1hw"
server_key: "c373b816848f37376bea0befd857dd1c"
server_key_iv: "77dea0264c070cb5d67f05fb32d98031"
server_key_version: "encryptionkey@0"
I20260926 16:43:09.695827 31873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance:
uuid: "677235c3ea774fe8b0a5a2b56c34fde0"
format_stamp: "Formatted at 2026-09-26 16:43:09 on dist-test-slave-n1hw"
server_key: "c373b816848f37376bea0befd857dd1c"
server_key_iv: "77dea0264c070cb5d67f05fb32d98031"
server_key_version: "encryptionkey@0"
I20260926 16:43:09.699488 31873 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 16:43:09.701884 31888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:09.703053 31873 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:43:09.703222 31873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/wal
uuid: "677235c3ea774fe8b0a5a2b56c34fde0"
format_stamp: "Formatted at 2026-09-26 16:43:09 on dist-test-slave-n1hw"
server_key: "c373b816848f37376bea0befd857dd1c"
server_key_iv: "77dea0264c070cb5d67f05fb32d98031"
server_key_version: "encryptionkey@0"
I20260926 16:43:09.703332 31873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:09.712275 31873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:09.714888 31873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:09.715077 31873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:09.715597 31873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:43:09.716339 31873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:43:09.716409 31873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:09.716471 31873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:43:09.716521 31873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:09.725028 31873 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.65:46501
I20260926 16:43:09.725071 32001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.65:46501 every 8 connection(s)
I20260926 16:43:09.725883 31873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
I20260926 16:43:09.731398 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 31873
I20260926 16:43:09.731523 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance
I20260926 16:43:09.731788 21553 external_mini_cluster.cc:1428] Setting key e959923caea51d1d41c021c5f27df736
Sep 26 16:43:09 dist-test-slave-n1hw krb5kdc[31766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440989, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:43:09.740850 31850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:09.727561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:53479 (local address 127.21.12.126:35049)
0926 16:43:09.727732 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:09.727737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:09.728505 (+ 768us) server_negotiation.cc:407] Connection header received
0926 16:43:09.729920 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:09.729923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:09.729969 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:09.730042 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:09.731114 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:09.731752 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:09.733172 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:09.733453 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:09.733537 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:09.736522 (+ 2985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:09.736531 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:09.736533 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:09.736555 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:09.738408 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:09.738918 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:09.738923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:09.738927 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:09.739029 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:09.739370 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:09.739375 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:09.739380 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:09.739627 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:09.739730 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:09.740457 (+ 727us) server_negotiation.cc:299] Negotiation successful
0926 16:43:09.740682 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 16:43:09.741788 32004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:09.727928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35049 (local address 127.21.12.65:53479)
0926 16:43:09.728353 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:09.728393 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:09.729643 (+ 1250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:09.730188 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:09.730194 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:09.730520 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:09.730986 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:09.730999 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:09.731882 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:09.731887 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:09.733048 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:09.733058 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:09.733610 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:09.733618 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:09.733862 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:09.734456 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:09.734480 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:09.736361 (+ 1881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:09.738540 (+ 2179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:09.738544 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:09.738555 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:09.738811 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:09.739134 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:09.739136 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:09.739138 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:09.739267 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:09.739791 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:09.739798 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:09.740106 (+ 308us) client_negotiation.cc:769] Sending connection context
0926 16:43:09.740334 (+ 228us) client_negotiation.cc:240] Negotiation successful
0926 16:43:09.740676 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for kudu/127.21.12.66@KRBTEST.COM; defaulting to no policy
I20260926 16:43:09.743108 32002 heartbeater.cc:347] Connected to a master server at 127.21.12.126:35049
I20260926 16:43:09.743410 32002 heartbeater.cc:464] Registering TS with master...
I20260926 16:43:09.743878 32002 heartbeater.cc:511] Master 127.21.12.126:35049 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.66@KRBTEST.COM" created.
I20260926 16:43:09.745476 31811 ts_manager.cc:194] Registered new tserver with Master: 677235c3ea774fe8b0a5a2b56c34fde0 (127.21.12.65:46501)
I20260926 16:43:09.746665 31811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.65@KRBTEST.COM'} at 127.21.12.65:53479
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab.
Entry for principal kudu/127.21.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab.
I20260926 16:43:09.761159 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.66:0
--local_ip_for_outbound_sockets=127.21.12.66
--webserver_interface=127.21.12.66
--webserver_port=0
--tserver_master_addrs=127.21.12.126:35049
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab
--principal=kudu/127.21.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:46439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:43:09.868291 32007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:09.868528 32007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:09.868569 32007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:09.872663 32007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:09.872718 32007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:09.872776 32007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.66
I20260926 16:43:09.877843 32007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:46439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab
--principal=kudu/127.21.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.66
--webserver_port=0
--tserver_master_addrs=127.21.12.126:35049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.66
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:43:09.878762 32007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:09.879693 32007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:09.889215 32013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:09.889225 32012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:09.889225 32015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:09.889380 32007 server_base.cc:1034] running on GCE node
I20260926 16:43:09.889940 32007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:09.890564 32007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:09.891728 32007 hybrid_clock.cc:648] HybridClock initialized: now 1790440989891712 us; error 26 us; skew 500 ppm
Sep 26 16:43:09 dist-test-slave-n1hw krb5kdc[31766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440989, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:43:09.894429 32007 init.cc:377] Logged in from keytab as kudu/127.21.12.66@KRBTEST.COM (short username kudu)
I20260926 16:43:09.895248 32007 webserver.cc:460] Webserver started at http://127.21.12.66:37219/ using document root <none> and password file <none>
I20260926 16:43:09.895793 32007 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:09.895926 32007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:09.896147 32007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:09.898289 32007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/data/instance:
uuid: "7bc961c7f2204aaea9c9e0c84ea2ec54"
format_stamp: "Formatted at 2026-09-26 16:43:09 on dist-test-slave-n1hw"
server_key: "f683a370e44818343f9bfea4ddbec817"
server_key_iv: "12eed0abe717524efa93faf15ccb2986"
server_key_version: "encryptionkey@0"
I20260926 16:43:09.898831 32007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance:
uuid: "7bc961c7f2204aaea9c9e0c84ea2ec54"
format_stamp: "Formatted at 2026-09-26 16:43:09 on dist-test-slave-n1hw"
server_key: "f683a370e44818343f9bfea4ddbec817"
server_key_iv: "12eed0abe717524efa93faf15ccb2986"
server_key_version: "encryptionkey@0"
I20260926 16:43:09.902467 32007 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:43:09.904822 32022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:09.906222 32007 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:43:09.906337 32007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/wal
uuid: "7bc961c7f2204aaea9c9e0c84ea2ec54"
format_stamp: "Formatted at 2026-09-26 16:43:09 on dist-test-slave-n1hw"
server_key: "f683a370e44818343f9bfea4ddbec817"
server_key_iv: "12eed0abe717524efa93faf15ccb2986"
server_key_version: "encryptionkey@0"
I20260926 16:43:09.906467 32007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:09.928157 32007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:09.931339 32007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:09.931536 32007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:09.932090 32007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:43:09.932869 32007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:43:09.932940 32007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:09.933004 32007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:43:09.933037 32007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:09.942020 32007 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.66:38989
I20260926 16:43:09.942061 32135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.66:38989 every 8 connection(s)
I20260926 16:43:09.942889 32007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
I20260926 16:43:09.947235 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 32007
I20260926 16:43:09.947326 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance
I20260926 16:43:09.947638 21553 external_mini_cluster.cc:1428] Setting key dca9895ace62321e15b1d48ef794e23d
Sep 26 16:43:09 dist-test-slave-n1hw krb5kdc[31766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440989, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.66@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.12.67@KRBTEST.COM; defaulting to no policy
I20260926 16:43:09.957137 31850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:09.944979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:41803 (local address 127.21.12.126:35049)
0926 16:43:09.945165 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:09.945170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:09.946320 (+ 1150us) server_negotiation.cc:407] Connection header received
0926 16:43:09.947923 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:09.947927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:09.947984 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:09.948062 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:09.949178 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:09.949724 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:09.950873 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:09.951120 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:09.951238 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:09.953550 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:09.953553 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:09.953556 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:09.953581 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:09.955009 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:09.955538 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:09.955541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:09.955543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:09.955623 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:09.956065 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:09.956068 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:09.956070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:09.956255 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:09.956375 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:09.956853 (+ 478us) server_negotiation.cc:299] Negotiation successful
0926 16:43:09.956978 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 16:43:09.958091 32138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:09.945369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35049 (local address 127.21.12.66:41803)
0926 16:43:09.946166 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:09.946221 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:09.947664 (+ 1443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:09.948231 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:09.948238 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:09.948571 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:09.949025 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:09.949038 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:09.949881 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:09.949884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:09.950740 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:09.950749 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:09.951269 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:09.951273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:09.951449 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:09.952014 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:09.952029 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:09.953390 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:09.955146 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:09.955152 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:09.955165 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:09.955423 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:09.955725 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:09.955728 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:09.955730 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:09.955906 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:09.956375 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:09.956383 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:09.956610 (+ 227us) client_negotiation.cc:769] Sending connection context
0926 16:43:09.956795 (+ 185us) client_negotiation.cc:240] Negotiation successful
0926 16:43:09.957201 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":125,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.67@KRBTEST.COM" created.
I20260926 16:43:09.959385 32136 heartbeater.cc:347] Connected to a master server at 127.21.12.126:35049
I20260926 16:43:09.959635 32136 heartbeater.cc:464] Registering TS with master...
I20260926 16:43:09.960196 32136 heartbeater.cc:511] Master 127.21.12.126:35049 requested a full tablet report, sending...
I20260926 16:43:09.961341 31811 ts_manager.cc:194] Registered new tserver with Master: 7bc961c7f2204aaea9c9e0c84ea2ec54 (127.21.12.66:38989)
I20260926 16:43:09.962291 31811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.66@KRBTEST.COM'} at 127.21.12.66:41803
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab.
Entry for principal kudu/127.21.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab.
I20260926 16:43:09.978184 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.67:0
--local_ip_for_outbound_sockets=127.21.12.67
--webserver_interface=127.21.12.67
--webserver_port=0
--tserver_master_addrs=127.21.12.126:35049
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab
--principal=kudu/127.21.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:46439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:43:10.088586 32141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:10.088871 32141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:10.088953 32141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:10.093223 32141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:10.093302 32141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:10.093387 32141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.67
I20260926 16:43:10.098593 32141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:46439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab
--principal=kudu/127.21.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.67
--webserver_port=0
--tserver_master_addrs=127.21.12.126:35049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.67
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:43:10.099614 32141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:10.100615 32141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:10.110791 32146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:10.110749 32147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:10.110837 32141 server_base.cc:1034] running on GCE node
W20260926 16:43:10.110764 32149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:10.111411 32141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:10.112120 32141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:10.113310 32141 hybrid_clock.cc:648] HybridClock initialized: now 1790440990113298 us; error 30 us; skew 500 ppm
Sep 26 16:43:10 dist-test-slave-n1hw krb5kdc[31766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440990, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:43:10.116159 32141 init.cc:377] Logged in from keytab as kudu/127.21.12.67@KRBTEST.COM (short username kudu)
I20260926 16:43:10.116997 32141 webserver.cc:460] Webserver started at http://127.21.12.67:45307/ using document root <none> and password file <none>
I20260926 16:43:10.117544 32141 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:10.117635 32141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:10.117919 32141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:10.119925 32141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/data/instance:
uuid: "251fdbdd3ec74865bebdcc61edb7d922"
format_stamp: "Formatted at 2026-09-26 16:43:10 on dist-test-slave-n1hw"
server_key: "e61a8cec26e6bd2724bc6644ca50deea"
server_key_iv: "a13e535d79fb354c3120d9591a64098a"
server_key_version: "encryptionkey@0"
I20260926 16:43:10.120491 32141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance:
uuid: "251fdbdd3ec74865bebdcc61edb7d922"
format_stamp: "Formatted at 2026-09-26 16:43:10 on dist-test-slave-n1hw"
server_key: "e61a8cec26e6bd2724bc6644ca50deea"
server_key_iv: "a13e535d79fb354c3120d9591a64098a"
server_key_version: "encryptionkey@0"
I20260926 16:43:10.124127 32141 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 16:43:10.126546 32156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:10.127792 32141 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:43:10.127962 32141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/wal
uuid: "251fdbdd3ec74865bebdcc61edb7d922"
format_stamp: "Formatted at 2026-09-26 16:43:10 on dist-test-slave-n1hw"
server_key: "e61a8cec26e6bd2724bc6644ca50deea"
server_key_iv: "a13e535d79fb354c3120d9591a64098a"
server_key_version: "encryptionkey@0"
I20260926 16:43:10.128077 32141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:10.153653 32141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:10.157109 32141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:10.157320 32141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:10.157997 32141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:43:10.158828 32141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:43:10.158900 32141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:10.158965 32141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:43:10.158998 32141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:10.168629 32141 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.67:44655
I20260926 16:43:10.168756 32269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.67:44655 every 8 connection(s)
I20260926 16:43:10.169623 32141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
I20260926 16:43:10.174500 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 32141
I20260926 16:43:10.174643 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance
I20260926 16:43:10.174937 21553 external_mini_cluster.cc:1428] Setting key cc30a6c60ccc970d0e964c6ee07af4c0
Sep 26 16:43:10 dist-test-slave-n1hw krb5kdc[31766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440990, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.67@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:43:10.183743 31850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:10.171572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:33541 (local address 127.21.12.126:35049)
0926 16:43:10.171711 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:10.171716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:10.172587 (+ 871us) server_negotiation.cc:407] Connection header received
0926 16:43:10.174083 (+ 1496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:10.174086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:10.174146 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:10.174221 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:10.175545 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:10.176080 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:10.177212 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:10.177455 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:10.177571 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:10.180106 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:10.180110 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:10.180113 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:10.180139 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:10.181677 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:10.182211 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:10.182216 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:10.182218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:10.182300 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:10.182659 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:10.182663 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:10.182665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:10.182845 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:10.182954 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:10.183451 (+ 497us) server_negotiation.cc:299] Negotiation successful
0926 16:43:10.183576 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 16:43:10.184530 32272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:10.171885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35049 (local address 127.21.12.67:33541)
0926 16:43:10.172442 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:10.172487 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:10.173856 (+ 1369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:10.174407 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:10.174414 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:10.174869 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:10.175397 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:10.175411 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:10.176211 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:10.176215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:10.177083 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:10.177092 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:10.177606 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:10.177610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:10.177834 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:10.178468 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:10.178484 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:10.179946 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:10.181819 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:10.181825 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:10.181840 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:10.182101 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:10.182413 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:10.182416 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:10.182418 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:10.182559 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:10.182963 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:10.182972 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:10.183215 (+ 243us) client_negotiation.cc:769] Sending connection context
0926 16:43:10.183407 (+ 192us) client_negotiation.cc:240] Negotiation successful
0926 16:43:10.183729 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":130,"threads_started":1}
I20260926 16:43:10.185635 32270 heartbeater.cc:347] Connected to a master server at 127.21.12.126:35049
I20260926 16:43:10.185918 32270 heartbeater.cc:464] Registering TS with master...
I20260926 16:43:10.186412 32270 heartbeater.cc:511] Master 127.21.12.126:35049 requested a full tablet report, sending...
I20260926 16:43:10.187480 31811 ts_manager.cc:194] Registered new tserver with Master: 251fdbdd3ec74865bebdcc61edb7d922 (127.21.12.67:44655)
I20260926 16:43:10.188143 31811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.67@KRBTEST.COM'} at 127.21.12.67:33541
I20260926 16:43:10.188666 21553 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:43:10.198906 31850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:10.190925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60862 (local address 127.21.12.126:35049)
0926 16:43:10.191052 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:10.191056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:10.191232 (+ 176us) server_negotiation.cc:407] Connection header received
0926 16:43:10.191469 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:10.191475 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:10.191534 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:10.191626 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:10.192707 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:10.193266 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:10.194215 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:10.194426 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:10.194505 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:10.195955 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:10.195958 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:10.195960 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:10.195980 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:10.197306 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:10.197722 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:10.197724 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:10.197725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:10.197858 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:10.198109 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:10.198111 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:10.198112 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:10.198320 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:10.198420 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:10.198666 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 16:43:10.198773 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 16:43:10.202294 31810 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60862:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:43:10.205258 31810 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:43:10.219677 32281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:10.214013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:46501 (local address 127.0.0.1:43582)
0926 16:43:10.214502 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:10.214554 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:10.214720 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:10.215391 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:10.215396 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:10.215421 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:43:10.215560 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:10.215570 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:10.216974 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:10.216980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:10.218273 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:10.218282 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:10.219119 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:10.219123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:10.219231 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:10.219284 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 16:43:10.219406 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 16:43:10.219531 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":1,"thread_start_us":91,"threads_started":1}
I20260926 16:43:10.219682 32282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:10.214117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43582 (local address 127.21.12.65:46501)
0926 16:43:10.214504 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:10.214523 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:10.214652 (+ 129us) server_negotiation.cc:407] Connection header received
0926 16:43:10.214994 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:10.215000 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:10.215222 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:10.215403 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:43:10.215717 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:10.216848 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:10.218398 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:10.218982 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:10.219126 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:10.219292 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:10.219436 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 16:43:10.219528 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":73,"threads_started":1}
I20260926 16:43:10.220644 32284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:10.215126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:38989 (local address 127.0.0.1:36476)
0926 16:43:10.215701 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:10.215715 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:10.215859 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:10.216534 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:10.216536 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:10.216552 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:43:10.216662 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:10.216668 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:10.218069 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:10.218072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:10.219165 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:10.219173 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:10.220268 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:10.220271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:10.220355 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:10.220375 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 16:43:10.220454 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 16:43:10.220521 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":83,"threads_started":1}
I20260926 16:43:10.220714 32285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:10.215321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36476 (local address 127.21.12.66:38989)
0926 16:43:10.215695 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:10.215707 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:10.215794 (+ 87us) server_negotiation.cc:407] Connection header received
0926 16:43:10.216028 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:10.216034 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:10.216233 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:10.216381 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:43:10.216800 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:10.217922 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:10.219290 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:10.220128 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:10.220215 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:10.220349 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:10.220486 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 16:43:10.220576 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":139,"threads_started":1}
I20260926 16:43:10.221163 32283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:10.214748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.67:44655 (local address 127.0.0.1:41642)
0926 16:43:10.215010 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:10.215022 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:10.215099 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:10.216274 (+ 1175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:10.216277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:10.216294 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:43:10.216383 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:10.216390 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:10.217688 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:10.217692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:10.218941 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:10.218951 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:10.220032 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:10.220036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:10.220126 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:10.220147 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 16:43:10.220956 (+ 809us) client_negotiation.cc:240] Negotiation successful
0926 16:43:10.221036 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":140,"threads_started":1}
I20260926 16:43:10.221388 32286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:10.215525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41642 (local address 127.21.12.67:44655)
0926 16:43:10.215882 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:10.215894 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:10.215915 (+ 21us) server_negotiation.cc:407] Connection header received
0926 16:43:10.215987 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:10.215992 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:10.216141 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:10.216293 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:43:10.216484 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:10.217545 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:10.219076 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:10.219900 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:10.220032 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:10.220162 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:10.221124 (+ 962us) server_negotiation.cc:299] Negotiation successful
0926 16:43:10.221242 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":129,"threads_started":1}
I20260926 16:43:10.223656 31936 tablet_service.cc:1467] Processing CreateTablet for tablet c1c7e37e8cac4ab487dee56bbdd0cb52 (DEFAULT_TABLE table=test-table [id=5074764f3d144f99af7a03deeda4fdeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:43:10.224761 31936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1c7e37e8cac4ab487dee56bbdd0cb52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:10.224603 32204 tablet_service.cc:1467] Processing CreateTablet for tablet c1c7e37e8cac4ab487dee56bbdd0cb52 (DEFAULT_TABLE table=test-table [id=5074764f3d144f99af7a03deeda4fdeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:43:10.224603 32070 tablet_service.cc:1467] Processing CreateTablet for tablet c1c7e37e8cac4ab487dee56bbdd0cb52 (DEFAULT_TABLE table=test-table [id=5074764f3d144f99af7a03deeda4fdeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:43:10.225584 32204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1c7e37e8cac4ab487dee56bbdd0cb52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:10.225579 32070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1c7e37e8cac4ab487dee56bbdd0cb52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:10.231007 32287 tablet_bootstrap.cc:492] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 677235c3ea774fe8b0a5a2b56c34fde0: Bootstrap starting.
I20260926 16:43:10.232409 32288 tablet_bootstrap.cc:492] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 7bc961c7f2204aaea9c9e0c84ea2ec54: Bootstrap starting.
I20260926 16:43:10.233436 32289 tablet_bootstrap.cc:492] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 251fdbdd3ec74865bebdcc61edb7d922: Bootstrap starting.
I20260926 16:43:10.233505 32287 tablet_bootstrap.cc:654] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 677235c3ea774fe8b0a5a2b56c34fde0: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:10.235160 32288 tablet_bootstrap.cc:654] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 7bc961c7f2204aaea9c9e0c84ea2ec54: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:10.235373 32289 tablet_bootstrap.cc:654] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 251fdbdd3ec74865bebdcc61edb7d922: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:10.235350 32287 log.cc:824] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 677235c3ea774fe8b0a5a2b56c34fde0: Log is configured to *not* fsync() on all Append() calls
I20260926 16:43:10.236003 32288 log.cc:824] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 7bc961c7f2204aaea9c9e0c84ea2ec54: Log is configured to *not* fsync() on all Append() calls
I20260926 16:43:10.236598 32289 log.cc:824] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 251fdbdd3ec74865bebdcc61edb7d922: Log is configured to *not* fsync() on all Append() calls
I20260926 16:43:10.237589 32287 tablet_bootstrap.cc:492] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 677235c3ea774fe8b0a5a2b56c34fde0: No bootstrap required, opened a new log
I20260926 16:43:10.237819 32287 ts_tablet_manager.cc:1397] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 677235c3ea774fe8b0a5a2b56c34fde0: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260926 16:43:10.238044 32288 tablet_bootstrap.cc:492] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 7bc961c7f2204aaea9c9e0c84ea2ec54: No bootstrap required, opened a new log
I20260926 16:43:10.238219 32288 ts_tablet_manager.cc:1397] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 7bc961c7f2204aaea9c9e0c84ea2ec54: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 16:43:10.238425 32289 tablet_bootstrap.cc:492] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 251fdbdd3ec74865bebdcc61edb7d922: No bootstrap required, opened a new log
I20260926 16:43:10.238612 32289 ts_tablet_manager.cc:1397] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 251fdbdd3ec74865bebdcc61edb7d922: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.001s
I20260926 16:43:10.241273 32289 raft_consensus.cc:348] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 251fdbdd3ec74865bebdcc61edb7d922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "677235c3ea774fe8b0a5a2b56c34fde0" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 46501 } } peers { permanent_uuid: "251fdbdd3ec74865bebdcc61edb7d922" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 44655 } } peers { permanent_uuid: "7bc961c7f2204aaea9c9e0c84ea2ec54" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 38989 } }
I20260926 16:43:10.241642 32289 raft_consensus.cc:374] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 251fdbdd3ec74865bebdcc61edb7d922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:10.241782 32289 raft_consensus.cc:729] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 251fdbdd3ec74865bebdcc61edb7d922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 251fdbdd3ec74865bebdcc61edb7d922, State: Initialized, Role: FOLLOWER
I20260926 16:43:10.241847 32288 raft_consensus.cc:348] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 7bc961c7f2204aaea9c9e0c84ea2ec54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "677235c3ea774fe8b0a5a2b56c34fde0" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 46501 } } peers { permanent_uuid: "251fdbdd3ec74865bebdcc61edb7d922" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 44655 } } peers { permanent_uuid: "7bc961c7f2204aaea9c9e0c84ea2ec54" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 38989 } }
I20260926 16:43:10.242156 32288 raft_consensus.cc:374] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 7bc961c7f2204aaea9c9e0c84ea2ec54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:10.241731 32287 raft_consensus.cc:348] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 677235c3ea774fe8b0a5a2b56c34fde0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "677235c3ea774fe8b0a5a2b56c34fde0" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 46501 } } peers { permanent_uuid: "251fdbdd3ec74865bebdcc61edb7d922" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 44655 } } peers { permanent_uuid: "7bc961c7f2204aaea9c9e0c84ea2ec54" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 38989 } }
I20260926 16:43:10.242254 32288 raft_consensus.cc:729] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 7bc961c7f2204aaea9c9e0c84ea2ec54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bc961c7f2204aaea9c9e0c84ea2ec54, State: Initialized, Role: FOLLOWER
I20260926 16:43:10.242340 32287 raft_consensus.cc:374] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 677235c3ea774fe8b0a5a2b56c34fde0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:10.242318 32289 consensus_queue.cc:260] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 251fdbdd3ec74865bebdcc61edb7d922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "677235c3ea774fe8b0a5a2b56c34fde0" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 46501 } } peers { permanent_uuid: "251fdbdd3ec74865bebdcc61edb7d922" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 44655 } } peers { permanent_uuid: "7bc961c7f2204aaea9c9e0c84ea2ec54" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 38989 } }
I20260926 16:43:10.242452 32287 raft_consensus.cc:729] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 677235c3ea774fe8b0a5a2b56c34fde0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 677235c3ea774fe8b0a5a2b56c34fde0, State: Initialized, Role: FOLLOWER
I20260926 16:43:10.242735 32288 consensus_queue.cc:260] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 7bc961c7f2204aaea9c9e0c84ea2ec54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "677235c3ea774fe8b0a5a2b56c34fde0" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 46501 } } peers { permanent_uuid: "251fdbdd3ec74865bebdcc61edb7d922" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 44655 } } peers { permanent_uuid: "7bc961c7f2204aaea9c9e0c84ea2ec54" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 38989 } }
I20260926 16:43:10.242849 32287 consensus_queue.cc:260] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 677235c3ea774fe8b0a5a2b56c34fde0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "677235c3ea774fe8b0a5a2b56c34fde0" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 46501 } } peers { permanent_uuid: "251fdbdd3ec74865bebdcc61edb7d922" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 44655 } } peers { permanent_uuid: "7bc961c7f2204aaea9c9e0c84ea2ec54" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 38989 } }
I20260926 16:43:10.243414 32289 ts_tablet_manager.cc:1428] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 251fdbdd3ec74865bebdcc61edb7d922: Time spent starting tablet: real 0.005s user 0.002s sys 0.002s
I20260926 16:43:10.243418 32136 heartbeater.cc:503] Master 127.21.12.126:35049 was elected leader, sending a full tablet report...
I20260926 16:43:10.243649 32288 ts_tablet_manager.cc:1428] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 7bc961c7f2204aaea9c9e0c84ea2ec54: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 16:43:10.243758 32270 heartbeater.cc:503] Master 127.21.12.126:35049 was elected leader, sending a full tablet report...
I20260926 16:43:10.244124 32287 ts_tablet_manager.cc:1428] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 677235c3ea774fe8b0a5a2b56c34fde0: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20260926 16:43:10.244421 32002 heartbeater.cc:503] Master 127.21.12.126:35049 was elected leader, sending a full tablet report...
I20260926 16:43:10.274338 32295 raft_consensus.cc:482] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 677235c3ea774fe8b0a5a2b56c34fde0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:43:10.274576 32295 raft_consensus.cc:504] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 677235c3ea774fe8b0a5a2b56c34fde0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "677235c3ea774fe8b0a5a2b56c34fde0" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 46501 } } peers { permanent_uuid: "251fdbdd3ec74865bebdcc61edb7d922" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 44655 } } peers { permanent_uuid: "7bc961c7f2204aaea9c9e0c84ea2ec54" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 38989 } }
I20260926 16:43:10.275695 32295 leader_election.cc:288] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 677235c3ea774fe8b0a5a2b56c34fde0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 251fdbdd3ec74865bebdcc61edb7d922 (127.21.12.67:44655), 7bc961c7f2204aaea9c9e0c84ea2ec54 (127.21.12.66:38989)
I20260926 16:43:10.279637 32286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:10.276033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:49763 (local address 127.21.12.67:44655)
0926 16:43:10.276200 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:10.276205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:10.276397 (+ 192us) server_negotiation.cc:407] Connection header received
0926 16:43:10.276580 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:10.276583 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:10.276646 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:10.276743 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:43:10.277041 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:10.277570 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:10.278717 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:10.279103 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:10.279237 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:10.279299 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:10.279458 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 16:43:10.279521 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 16:43:10.279654 32296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:10.275919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.67:44655 (local address 127.21.12.65:49763)
0926 16:43:10.276285 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:10.276299 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:10.276451 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:10.276763 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:10.276766 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:10.276785 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:43:10.276916 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:10.276922 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:10.277686 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:10.277691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:10.278576 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:10.278585 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:10.279227 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:10.279231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:10.279334 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:10.279358 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 16:43:10.279429 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 16:43:10.279507 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":177,"threads_started":1}
I20260926 16:43:10.279784 32285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:10.276555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:59363 (local address 127.21.12.66:38989)
0926 16:43:10.276678 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:10.276684 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:10.276722 (+ 38us) server_negotiation.cc:407] Connection header received
0926 16:43:10.276899 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:10.276902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:10.276945 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:10.277033 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:43:10.277290 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:10.277811 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:10.278822 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:10.279223 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:10.279354 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:10.279405 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:10.279604 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 16:43:10.279671 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 16:43:10.279804 32297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:10.276408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:38989 (local address 127.21.12.65:59363)
0926 16:43:10.276660 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:10.276672 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:10.276755 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:10.277046 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:10.277048 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:10.277064 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:43:10.277181 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:10.277186 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:10.277930 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:10.277936 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:10.278692 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:10.278701 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:10.279348 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:10.279351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:10.279481 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:10.279499 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 16:43:10.279590 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 16:43:10.279660 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":94,"threads_started":1}
I20260926 16:43:10.280387 32090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1c7e37e8cac4ab487dee56bbdd0cb52" candidate_uuid: "677235c3ea774fe8b0a5a2b56c34fde0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bc961c7f2204aaea9c9e0c84ea2ec54" is_pre_election: true
I20260926 16:43:10.280548 32224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1c7e37e8cac4ab487dee56bbdd0cb52" candidate_uuid: "677235c3ea774fe8b0a5a2b56c34fde0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "251fdbdd3ec74865bebdcc61edb7d922" is_pre_election: true
I20260926 16:43:10.280689 32090 raft_consensus.cc:2454] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 7bc961c7f2204aaea9c9e0c84ea2ec54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 677235c3ea774fe8b0a5a2b56c34fde0 in term 0.
I20260926 16:43:10.280920 32224 raft_consensus.cc:2454] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 251fdbdd3ec74865bebdcc61edb7d922 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 677235c3ea774fe8b0a5a2b56c34fde0 in term 0.
I20260926 16:43:10.281216 31890 leader_election.cc:302] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 677235c3ea774fe8b0a5a2b56c34fde0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 677235c3ea774fe8b0a5a2b56c34fde0, 7bc961c7f2204aaea9c9e0c84ea2ec54; no voters:
I20260926 16:43:10.281473 32295 raft_consensus.cc:2781] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 677235c3ea774fe8b0a5a2b56c34fde0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:43:10.281548 32295 raft_consensus.cc:482] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 677235c3ea774fe8b0a5a2b56c34fde0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:43:10.281579 32295 raft_consensus.cc:3037] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 677235c3ea774fe8b0a5a2b56c34fde0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:10.282660 32295 raft_consensus.cc:504] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 677235c3ea774fe8b0a5a2b56c34fde0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "677235c3ea774fe8b0a5a2b56c34fde0" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 46501 } } peers { permanent_uuid: "251fdbdd3ec74865bebdcc61edb7d922" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 44655 } } peers { permanent_uuid: "7bc961c7f2204aaea9c9e0c84ea2ec54" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 38989 } }
I20260926 16:43:10.282996 32295 leader_election.cc:288] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 677235c3ea774fe8b0a5a2b56c34fde0 [CANDIDATE]: Term 1 election: Requested vote from peers 251fdbdd3ec74865bebdcc61edb7d922 (127.21.12.67:44655), 7bc961c7f2204aaea9c9e0c84ea2ec54 (127.21.12.66:38989)
I20260926 16:43:10.283372 32224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1c7e37e8cac4ab487dee56bbdd0cb52" candidate_uuid: "677235c3ea774fe8b0a5a2b56c34fde0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "251fdbdd3ec74865bebdcc61edb7d922"
I20260926 16:43:10.283485 32090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1c7e37e8cac4ab487dee56bbdd0cb52" candidate_uuid: "677235c3ea774fe8b0a5a2b56c34fde0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bc961c7f2204aaea9c9e0c84ea2ec54"
I20260926 16:43:10.283522 32224 raft_consensus.cc:3037] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 251fdbdd3ec74865bebdcc61edb7d922 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:10.283629 32090 raft_consensus.cc:3037] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 7bc961c7f2204aaea9c9e0c84ea2ec54 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:10.284610 32224 raft_consensus.cc:2454] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 251fdbdd3ec74865bebdcc61edb7d922 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 677235c3ea774fe8b0a5a2b56c34fde0 in term 1.
I20260926 16:43:10.284674 32090 raft_consensus.cc:2454] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 7bc961c7f2204aaea9c9e0c84ea2ec54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 677235c3ea774fe8b0a5a2b56c34fde0 in term 1.
I20260926 16:43:10.284895 31890 leader_election.cc:302] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 677235c3ea774fe8b0a5a2b56c34fde0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 251fdbdd3ec74865bebdcc61edb7d922, 677235c3ea774fe8b0a5a2b56c34fde0; no voters:
I20260926 16:43:10.285069 32295 raft_consensus.cc:2781] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 677235c3ea774fe8b0a5a2b56c34fde0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:43:10.285297 32295 raft_consensus.cc:686] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 677235c3ea774fe8b0a5a2b56c34fde0 [term 1 LEADER]: Becoming Leader. State: Replica: 677235c3ea774fe8b0a5a2b56c34fde0, State: Running, Role: LEADER
I20260926 16:43:10.285653 32295 consensus_queue.cc:237] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 677235c3ea774fe8b0a5a2b56c34fde0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "677235c3ea774fe8b0a5a2b56c34fde0" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 46501 } } peers { permanent_uuid: "251fdbdd3ec74865bebdcc61edb7d922" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 44655 } } peers { permanent_uuid: "7bc961c7f2204aaea9c9e0c84ea2ec54" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 38989 } }
I20260926 16:43:10.288630 31810 catalog_manager.cc:5495] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 677235c3ea774fe8b0a5a2b56c34fde0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 677235c3ea774fe8b0a5a2b56c34fde0 (127.21.12.65). New cstate: current_term: 1 leader_uuid: "677235c3ea774fe8b0a5a2b56c34fde0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "677235c3ea774fe8b0a5a2b56c34fde0" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 46501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "251fdbdd3ec74865bebdcc61edb7d922" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 44655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bc961c7f2204aaea9c9e0c84ea2ec54" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 38989 } health_report { overall_health: UNKNOWN } } }
I20260926 16:43:10.316943 32282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:10.313423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43594 (local address 127.21.12.65:46501)
0926 16:43:10.313553 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:10.313557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:10.313571 (+ 14us) server_negotiation.cc:407] Connection header received
0926 16:43:10.313643 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:10.313645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:10.313697 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:10.313879 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:43:10.314076 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:10.314557 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:10.315413 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:10.315628 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:10.315714 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:10.315981 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:43:10.316382 (+ 401us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:43:10.316582 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:10.316685 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 16:43:10.316735 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 16:43:10.322654 32090 raft_consensus.cc:1261] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 7bc961c7f2204aaea9c9e0c84ea2ec54 [term 1 FOLLOWER]: Refusing update from remote peer 677235c3ea774fe8b0a5a2b56c34fde0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:43:10.322652 32224 raft_consensus.cc:1261] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 251fdbdd3ec74865bebdcc61edb7d922 [term 1 FOLLOWER]: Refusing update from remote peer 677235c3ea774fe8b0a5a2b56c34fde0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:43:10.323370 32298 consensus_queue.cc:1035] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 677235c3ea774fe8b0a5a2b56c34fde0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bc961c7f2204aaea9c9e0c84ea2ec54" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:43:10.323557 32295 consensus_queue.cc:1035] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 677235c3ea774fe8b0a5a2b56c34fde0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "251fdbdd3ec74865bebdcc61edb7d922" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 44655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:43:10.330677 32301 mvcc.cc:205] Tried to move back new op lower bound from 7333646296354537472 to 7333646296212848640. Current Snapshot: MvccSnapshot[applied={T|T < 7333646296354537472}]
I20260926 16:43:10.332561 32303 mvcc.cc:205] Tried to move back new op lower bound from 7333646296354537472 to 7333646296212848640. Current Snapshot: MvccSnapshot[applied={T|T < 7333646296354537472}]
I20260926 16:43:10.339288 31811 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:43:10.339485 31811 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:43:10.341872 31811 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f5319cda62874eb789beb86cc8248b18: Sending DeleteTablet for 3 replicas of tablet c1c7e37e8cac4ab487dee56bbdd0cb52
I20260926 16:43:10.342587 31936 tablet_service.cc:1514] Processing DeleteTablet for tablet c1c7e37e8cac4ab487dee56bbdd0cb52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:43:10 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:43582
I20260926 16:43:10.342742 32204 tablet_service.cc:1514] Processing DeleteTablet for tablet c1c7e37e8cac4ab487dee56bbdd0cb52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:43:10 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:41642
I20260926 16:43:10.343083 32070 tablet_service.cc:1514] Processing DeleteTablet for tablet c1c7e37e8cac4ab487dee56bbdd0cb52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:43:10 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:36476
I20260926 16:43:10.343163 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 31873
I20260926 16:43:10.343806 32311 tablet_replica.cc:330] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 251fdbdd3ec74865bebdcc61edb7d922: stopping tablet replica
I20260926 16:43:10.344094 32311 raft_consensus.cc:2229] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 251fdbdd3ec74865bebdcc61edb7d922 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:43:10.344379 32311 raft_consensus.cc:2258] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 251fdbdd3ec74865bebdcc61edb7d922 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:43:10.345927 32311 ts_tablet_manager.cc:1905] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 251fdbdd3ec74865bebdcc61edb7d922: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:43:10.348256 32311 ts_tablet_manager.cc:1918] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 251fdbdd3ec74865bebdcc61edb7d922: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:43:10.348344 32311 log.cc:1196] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 251fdbdd3ec74865bebdcc61edb7d922: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440924937351-21553-0/minicluster-data/ts-2/wal/wals/c1c7e37e8cac4ab487dee56bbdd0cb52
I20260926 16:43:10.348573 32312 tablet_replica.cc:330] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 7bc961c7f2204aaea9c9e0c84ea2ec54: stopping tablet replica
I20260926 16:43:10.348711 32311 ts_tablet_manager.cc:1939] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 251fdbdd3ec74865bebdcc61edb7d922: Deleting consensus metadata
I20260926 16:43:10.348914 32312 raft_consensus.cc:2229] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 7bc961c7f2204aaea9c9e0c84ea2ec54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:43:10.349226 32312 raft_consensus.cc:2258] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 7bc961c7f2204aaea9c9e0c84ea2ec54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:43:10.350128 31796 catalog_manager.cc:4841] TS 251fdbdd3ec74865bebdcc61edb7d922 (127.21.12.67:44655): tablet c1c7e37e8cac4ab487dee56bbdd0cb52 (table test-table [id=5074764f3d144f99af7a03deeda4fdeb]) successfully deleted
W20260926 16:43:10.350854 31796 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.12.65:46501 (error 108)
I20260926 16:43:10.350945 32312 ts_tablet_manager.cc:1905] T c1c7e37e8cac4ab487dee56bbdd0cb52 P 7bc961c7f2204aaea9c9e0c84ea2ec54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:43:10.351001 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 32007
I20260926 16:43:10.352623 32283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:10.352257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:46501 (local address 127.0.0.1:43600)
0926 16:43:10.352432 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:10.352514 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.65:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260926 16:43:10.352809 31796 catalog_manager.cc:4584] TS 677235c3ea774fe8b0a5a2b56c34fde0 (127.21.12.65:46501): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c1c7e37e8cac4ab487dee56bbdd0cb52: Network error: Client connection negotiation failed: client connection to 127.21.12.65:46501: connect: Connection refused (error 111)
I20260926 16:43:10.358060 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 32141
I20260926 16:43:10.359170 32283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:10.358362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:38989 (local address 127.0.0.1:36492)
0926 16:43:10.358962 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:10.359034 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.66:38989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450}
I20260926 16:43:10.363888 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 31779
2026-09-26T16:43:10Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2283 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:43:10 dist-test-slave-n1hw krb5kdc[32316](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:43:10 dist-test-slave-n1hw krb5kdc[32316](info): set up 2 sockets
Sep 26 16:43:10 dist-test-slave-n1hw krb5kdc[32316](info): commencing operation
krb5kdc: starting...
W20260926 16:43:11.412823 21553 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Sep 26 16:43:11 dist-test-slave-n1hw krb5kdc[32316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:43:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:43:11Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.21.12.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.21.12.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.21.12.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/krb5kdc/rangeradmin_127.21.12.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.21.12.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/krb5kdc/rangeradmin_127.21.12.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.21.12.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.21.12.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.21.12.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/krb5kdc/rangerlookup_127.21.12.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.21.12.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/krb5kdc/rangerlookup_127.21.12.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.12.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.21.12.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.12.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/krb5kdc/HTTP_127.21.12.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.12.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/krb5kdc/HTTP_127.21.12.84@KRBTEST.COM.keytab.
I20260926 16:43:11.590687 21553 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskEZzU7d/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/postgres -l logfile start
2026-09-26 16:43:14.243 UTC [32346] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 16:43:14.244 UTC [32346] LOG: listening on IPv4 address "127.21.12.84", port 39671
2026-09-26 16:43:14.248 UTC [32346] LOG: listening on Unix socket "/tmp/.s.PGSQL.39671"
2026-09-26 16:43:14.282 UTC [32349] LOG: database system was shut down at 2026-09-26 16:43:12 UTC
2026-09-26 16:43:14.286 UTC [32346] LOG: database system is ready to accept connections
I20260926 16:43:15.224615 21553 mini_postgres.cc:96] Postgres bound to 39671
2026-09-26 16:43:15.231 UTC [32357] FATAL: database "slave" does not exist
127.21.12.84:39671 - accepting connections
I20260926 16:43:15.232151 21553 mini_ranger.cc:161] Starting Ranger...
I20260926 16:43:15.247427 21553 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 16:43:15.824705 21553 mini_ranger.cc:87] Created ranger Postgres database
I20260926 16:43:15.824887 21553 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ranger-admin
2026-09-26 16:43:16,144 [I] DB FLAVOR :POSTGRES
2026-09-26 16:43:16,144 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 16:43:16,144 [I] Checking connection..
2026-09-26 16:43:16,145 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.84:39671/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 16:43:16,416 [I] Checking connection passed.
2026-09-26 16:43:16,416 [I] --------- Verifying version history table ---------
2026-09-26 16:43:16,416 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.84:39671/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 16:43:16,670 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 16:43:16,671 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.84:39671/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 16:43:16,926 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 16:43:16,926 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 16:43:16,927 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.84:39671/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 16:43:17.170 UTC [32421] WARNING: there is no transaction in progress
2026-09-26 16:43:17,176 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 16:43:17,176 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.84:39671/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 16:43:17,451 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 16:43:17,452 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 16:43:17,452 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.84:39671/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 16:43:17,673 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.84:39671/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 16:43:17,899 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.84:39671/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-n1hw.c.gcp-upstream.internal','N') ;"
2026-09-26 16:43:18,135 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.84:39671/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 16:43:18,401 [I] Table x_portal_user does not exist in database ranger
2026-09-26 16:43:18,401 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.84:39671/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 16:43:18,636 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 16:43:18,637 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.84:39671/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 16:43:18,869 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.84:39671/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 16:43:19,108 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 16:43:19,109 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.84:39671/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 16:43:20.103 UTC [32529] WARNING: there is no transaction in progress
2026-09-26 16:43:20.120 UTC [32529] WARNING: there is no transaction in progress
2026-09-26 16:43:20.137 UTC [32529] WARNING: there is no transaction in progress
2026-09-26 16:43:20.154 UTC [32529] WARNING: there is no transaction in progress
2026-09-26 16:43:20.172 UTC [32529] WARNING: there is no transaction in progress
2026-09-26 16:43:20.192 UTC [32529] WARNING: there is no transaction in progress
2026-09-26 16:43:20.249 UTC [32529] WARNING: there is no transaction in progress
2026-09-26 16:43:20.256 UTC [32529] WARNING: there is no transaction in progress
2026-09-26 16:43:20.270 UTC [32529] WARNING: there is no transaction in progress
2026-09-26 16:43:20.281 UTC [32529] WARNING: there is no transaction in progress
2026-09-26 16:43:20.292 UTC [32529] WARNING: there is no transaction in progress
2026-09-26 16:43:20.308 UTC [32529] WARNING: there is no transaction in progress
2026-09-26 16:43:20.318 UTC [32529] WARNING: there is no transaction in progress
2026-09-26 16:43:20.324 UTC [32529] WARNING: there is no transaction in progress
2026-09-26 16:43:20.332 UTC [32529] WARNING: there is no transaction in progress
2026-09-26 16:43:20.339 UTC [32529] WARNING: there is no transaction in progress
2026-09-26 16:43:20,993 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 16:43:20,994 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.84:39671/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 16:43:21,225 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 16:43:21,226 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.84:39671/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 16:43:21,463 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 16:43:21,463 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.84:39671/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 16:43:21,704 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 16:43:21,704 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.84:39671/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 16:43:21,955 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.84:39671/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 16:43:22,196 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.84:39671/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-n1hw.c.gcp-upstream.internal';"
2026-09-26 16:43:22,442 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 16:43:22,442 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.84:39671/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 16:43:22,689 [I] DB_PATCHES have already been applied
I20260926 16:43:22.695611 21553 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ranger-admin:/tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskEZzU7d/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 16:43:22.695680 21553 mini_ranger.cc:193] Using host: 127.21.12.84
I20260926 16:43:22.699193 21553 mini_ranger.cc:227] Ranger admin URL: http://127.21.12.84:44523
Sep 26, 2026 4:43:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 4:43:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 4:43:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 4:43:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 4:43:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 4:43:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 4:43:23 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 16:43:23.819289 21553 mini_ranger.cc:160] Time spent starting Ranger: real 8.587s user 0.003s sys 0.007s
Sep 26, 2026 4:43:42 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 4:43:42 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 4:43:42 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 4:43:42 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 4:43:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 4:43:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 4:43:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 4:43:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 4:43:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 4:43:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 4:43:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 4:43:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 4:43:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 4:43:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 4:43:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 4:43:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 4:43:42 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 4:43:43 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 16:43:44.316207 21553 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.21.12.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.21.12.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.21.12.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/krb5kdc/rangerkms_127.21.12.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.21.12.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/krb5kdc/rangerkms_127.21.12.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.12.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.21.12.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.12.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/krb5kdc/HTTP_127.21.12.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.12.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/krb5kdc/HTTP_127.21.12.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 16:43:44 dist-test-slave-n1hw krb5kdc[32316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441024, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 16:43:44.406205 21553 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 16:43:44.416527 21553 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 16:43:45.314826 21553 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 16:43:45.314926 21553 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ranger-kms
I20260926 16:43:45.315029 21553 mini_ranger_kms.cc:227] Using postgres at 127.21.12.84:39671
2026-09-26 16:43:45,499 [I] DB FLAVOR :POSTGRES
2026-09-26 16:43:45,499 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 16:43:45,499 [I] Checking connection
2026-09-26 16:43:45,499 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.84:39671/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 16:43:45,771 [I] connection success
2026-09-26 16:43:45,771 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 16:43:45,771 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 16:43:45,771 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.84:39671/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 16:43:46,039 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 16:43:46,039 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 16:43:46,039 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 16:43:46,040 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.84:39671/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 16:43:46,329 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 16:43:46.790453 21553 mini_ranger_kms.cc:310] Created kms service
I20260926 16:43:47.005574 21553 mini_ranger_kms.cc:326] Created kudu user
I20260926 16:43:47.050457 21553 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 16:43:47 dist-test-slave-n1hw krb5kdc[32316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441024, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.21.12.84@KRBTEST.COM
I20260926 16:43:47.492002 21553 mini_ranger_kms.cc:384] Added ranger policy
I20260926 16:43:47.492177 21553 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ranger-kms:/tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 16:43:47.492202 21553 mini_ranger_kms.cc:240] Using host: 127.21.12.84
I20260926 16:43:47.496340 21553 mini_ranger_kms.cc:276] Ranger KMS PID: 32709
I20260926 16:43:47.496482 21553 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.21.12.84:55169
Sep 26, 2026 4:43:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 4:43:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 4:43:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 4:43:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 4:43:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 4:43:49 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 16:43:49.800897 21553 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.395s user 0.008s sys 0.160s
I20260926 16:43:49.801020 21553 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 16:43:49.801138 21553 mini_ranger_kms.cc:401] 127.21.12.84:55169/kms/v1/keys
Sep 26 16:43:49 dist-test-slave-n1hw krb5kdc[32316](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790441029, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.12.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 16:43:50 dist-test-slave-n1hw krb5kdc[32316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441029, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.12.84@KRBTEST.COM for HTTP/127.21.12.84@KRBTEST.COM
Sep 26 16:43:50 dist-test-slave-n1hw krb5kdc[32316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441029, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.12.84@KRBTEST.COM for HTTP/127.21.12.84@KRBTEST.COM
Sep 26 16:43:52 dist-test-slave-n1hw krb5kdc[32316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.21.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
I20260926 16:43:52.961330 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:39655
--webserver_interface=127.21.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:43945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:39655
--ranger_config_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.12.84:55169/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:43:53.070663 32760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:53.070926 32760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:53.070966 32760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:53.074995 32760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:43:53.075053 32760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:53.075073 32760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:53.075089 32760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:43:53.075104 32760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:43:53.080762 32760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:43945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/master-0/wal
--ranger_kms_url=127.21.12.84:55169/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:39655
--ranger_config_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:39655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:43:53.081893 32760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:53.082937 32760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:53.091336 32765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:53.091346 32766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:53.091485 32760 server_base.cc:1034] running on GCE node
W20260926 16:43:53.091346 300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:53.092293 32760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:53.093578 32760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:53.094779 32760 hybrid_clock.cc:648] HybridClock initialized: now 1790441033094765 us; error 55 us; skew 500 ppm
Sep 26 16:43:53 dist-test-slave-n1hw krb5kdc[32316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441033, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:43:53.103029 32760 init.cc:377] Logged in from keytab as kudu/127.21.12.126@KRBTEST.COM (short username kudu)
I20260926 16:43:53.103984 32760 webserver.cc:460] Webserver started at http://127.21.12.126:42239/ using document root <none> and password file <none>
I20260926 16:43:53.104511 32760 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:53.104563 32760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:53.104777 32760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 16:43:53 dist-test-slave-n1hw krb5kdc[32316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441033, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for HTTP/127.21.12.84@KRBTEST.COM
I20260926 16:43:53.169170 32760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "8e7fa2bbcd49447ab16d1389884a2d42"
format_stamp: "Formatted at 2026-09-26 16:43:53 on dist-test-slave-n1hw"
server_key: "7bf3a7ca4bc2586a516ee8309d2e38f8"
server_key_iv: "087570cbae41a057a16f5e7d8a82fe4d"
server_key_version: "kuduclusterkey@0"
I20260926 16:43:53.169849 32760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "8e7fa2bbcd49447ab16d1389884a2d42"
format_stamp: "Formatted at 2026-09-26 16:43:53 on dist-test-slave-n1hw"
server_key: "7bf3a7ca4bc2586a516ee8309d2e38f8"
server_key_iv: "087570cbae41a057a16f5e7d8a82fe4d"
server_key_version: "kuduclusterkey@0"
I20260926 16:43:53.174026 32760 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260926 16:43:53.190286 307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:53.191947 32760 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:43:53.192090 32760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "8e7fa2bbcd49447ab16d1389884a2d42"
format_stamp: "Formatted at 2026-09-26 16:43:53 on dist-test-slave-n1hw"
server_key: "7bf3a7ca4bc2586a516ee8309d2e38f8"
server_key_iv: "087570cbae41a057a16f5e7d8a82fe4d"
server_key_version: "kuduclusterkey@0"
I20260926 16:43:53.192266 32760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:53.214373 32760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:53.217840 32760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:53.218052 32760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:53.227186 32760 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.126:39655
I20260926 16:43:53.227247 359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.126:39655 every 8 connection(s)
I20260926 16:43:53.228515 32760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:43:53.229084 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 32760
I20260926 16:43:53.229230 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:43:53.229506 21553 external_mini_cluster.cc:1428] Setting key 51d98de061e872407b44c21ab70412d2
I20260926 16:43:53.232324 360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 16:43:53 dist-test-slave-n1hw krb5kdc[32316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:43:53.239575 360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e7fa2bbcd49447ab16d1389884a2d42: Bootstrap starting.
I20260926 16:43:53.242404 360 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e7fa2bbcd49447ab16d1389884a2d42: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:53.243584 363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:53.231009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51420 (local address 127.21.12.126:39655)
0926 16:43:53.231450 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:53.231463 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:53.231497 (+ 34us) server_negotiation.cc:407] Connection header received
0926 16:43:53.232128 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:53.232144 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:53.232521 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:53.232866 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:53.233644 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:53.234685 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:53.235541 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:53.236003 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:53.236170 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:53.238865 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:53.238870 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:53.238882 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:53.238907 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:53.240737 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:53.241169 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:53.241175 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:53.241182 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:53.241302 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:53.241603 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:53.241606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:53.241608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:53.242067 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:53.242267 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:53.242553 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 16:43:53.242923 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":115,"threads_started":1}
I20260926 16:43:53.243913 360 log.cc:824] T 00000000000000000000000000000000 P 8e7fa2bbcd49447ab16d1389884a2d42: Log is configured to *not* fsync() on all Append() calls
I20260926 16:43:53.247219 360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e7fa2bbcd49447ab16d1389884a2d42: No bootstrap required, opened a new log
I20260926 16:43:53.251507 360 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e7fa2bbcd49447ab16d1389884a2d42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e7fa2bbcd49447ab16d1389884a2d42" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 39655 } }
I20260926 16:43:53.251783 360 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e7fa2bbcd49447ab16d1389884a2d42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:53.251843 360 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e7fa2bbcd49447ab16d1389884a2d42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7fa2bbcd49447ab16d1389884a2d42, State: Initialized, Role: FOLLOWER
I20260926 16:43:53.252313 360 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e7fa2bbcd49447ab16d1389884a2d42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e7fa2bbcd49447ab16d1389884a2d42" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 39655 } }
I20260926 16:43:53.252465 360 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e7fa2bbcd49447ab16d1389884a2d42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:43:53.252527 360 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e7fa2bbcd49447ab16d1389884a2d42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:43:53.252622 360 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e7fa2bbcd49447ab16d1389884a2d42 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:53.253732 360 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e7fa2bbcd49447ab16d1389884a2d42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e7fa2bbcd49447ab16d1389884a2d42" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 39655 } }
I20260926 16:43:53.254112 360 leader_election.cc:302] T 00000000000000000000000000000000 P 8e7fa2bbcd49447ab16d1389884a2d42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e7fa2bbcd49447ab16d1389884a2d42; no voters:
I20260926 16:43:53.254462 360 leader_election.cc:288] T 00000000000000000000000000000000 P 8e7fa2bbcd49447ab16d1389884a2d42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:43:53.254607 365 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e7fa2bbcd49447ab16d1389884a2d42 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:43:53.254869 365 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e7fa2bbcd49447ab16d1389884a2d42 [term 1 LEADER]: Becoming Leader. State: Replica: 8e7fa2bbcd49447ab16d1389884a2d42, State: Running, Role: LEADER
I20260926 16:43:53.255234 365 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e7fa2bbcd49447ab16d1389884a2d42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e7fa2bbcd49447ab16d1389884a2d42" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 39655 } }
I20260926 16:43:53.255761 360 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e7fa2bbcd49447ab16d1389884a2d42 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:43:53.257223 367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e7fa2bbcd49447ab16d1389884a2d42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e7fa2bbcd49447ab16d1389884a2d42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e7fa2bbcd49447ab16d1389884a2d42" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 39655 } } }
I20260926 16:43:53.256562 365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e7fa2bbcd49447ab16d1389884a2d42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e7fa2bbcd49447ab16d1389884a2d42. Latest consensus state: current_term: 1 leader_uuid: "8e7fa2bbcd49447ab16d1389884a2d42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e7fa2bbcd49447ab16d1389884a2d42" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 39655 } } }
I20260926 16:43:53.257457 367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e7fa2bbcd49447ab16d1389884a2d42 [sys.catalog]: This master's current role is: LEADER
I20260926 16:43:53.257481 365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e7fa2bbcd49447ab16d1389884a2d42 [sys.catalog]: This master's current role is: LEADER
I20260926 16:43:53.257951 379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:43:53.258018 360 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 16:43:53.264500 379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:43:53.277011 379 catalog_manager.cc:1347] Generated new cluster ID: be408e47860e408eb1aa74d37946ff80
I20260926 16:43:53.277077 379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:43:53.295277 379 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:43:53.296620 379 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:43:53.307677 379 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e7fa2bbcd49447ab16d1389884a2d42: Generated new TSK 0
I20260926 16:43:53.308714 379 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab.
Entry for principal kudu/127.21.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab.
I20260926 16:43:54.078997 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.12.84:55169/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.65:0
--local_ip_for_outbound_sockets=127.21.12.65
--webserver_interface=127.21.12.65
--webserver_port=0
--tserver_master_addrs=127.21.12.126:39655
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab
--principal=kudu/127.21.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:43945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:43:54.249228 405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:54.250036 405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:54.250131 405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:54.256266 405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:54.256342 405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:54.256449 405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.65
I20260926 16:43:54.264494 405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:43945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.21.12.84:55169/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab
--principal=kudu/127.21.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.65
--webserver_port=0
--tserver_master_addrs=127.21.12.126:39655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.65
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:43:54.265902 405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:54.267295 405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:54.278329 413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:54.278332 412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:54.278694 415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:54.278822 405 server_base.cc:1034] running on GCE node
I20260926 16:43:54.279378 405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:54.280176 405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:54.281426 405 hybrid_clock.cc:648] HybridClock initialized: now 1790441034281401 us; error 43 us; skew 500 ppm
Sep 26 16:43:54 dist-test-slave-n1hw krb5kdc[32316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441034, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:43:54.284346 405 init.cc:377] Logged in from keytab as kudu/127.21.12.65@KRBTEST.COM (short username kudu)
I20260926 16:43:54.285409 405 webserver.cc:460] Webserver started at http://127.21.12.65:42117/ using document root <none> and password file <none>
I20260926 16:43:54.286099 405 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:54.286188 405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:54.286440 405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 16:43:54 dist-test-slave-n1hw krb5kdc[32316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441034, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for HTTP/127.21.12.84@KRBTEST.COM
I20260926 16:43:54.298853 405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-0/data/instance:
uuid: "2cdbd3b405fc4dd3b1a2502c7aad9df8"
format_stamp: "Formatted at 2026-09-26 16:43:54 on dist-test-slave-n1hw"
server_key: "9b08a0a1f26cf01dc4baa55220b3b108"
server_key_iv: "47a38d3626fb5e5e89f79610e9b1550a"
server_key_version: "kuduclusterkey@0"
I20260926 16:43:54.299589 405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance:
uuid: "2cdbd3b405fc4dd3b1a2502c7aad9df8"
format_stamp: "Formatted at 2026-09-26 16:43:54 on dist-test-slave-n1hw"
server_key: "9b08a0a1f26cf01dc4baa55220b3b108"
server_key_iv: "47a38d3626fb5e5e89f79610e9b1550a"
server_key_version: "kuduclusterkey@0"
I20260926 16:43:54.304942 405 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:43:54.342047 422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:54.343824 405 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.003s sys 0.000s
I20260926 16:43:54.343971 405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-0/wal
uuid: "2cdbd3b405fc4dd3b1a2502c7aad9df8"
format_stamp: "Formatted at 2026-09-26 16:43:54 on dist-test-slave-n1hw"
server_key: "9b08a0a1f26cf01dc4baa55220b3b108"
server_key_iv: "47a38d3626fb5e5e89f79610e9b1550a"
server_key_version: "kuduclusterkey@0"
I20260926 16:43:54.344125 405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:54.363267 405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:54.366560 405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:54.366761 405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:54.367422 405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:43:54.368436 405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:43:54.368495 405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:54.368561 405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:43:54.368634 405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:54.378516 405 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.65:45471
I20260926 16:43:54.379755 405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
I20260926 16:43:54.382632 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 405
I20260926 16:43:54.382777 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance
I20260926 16:43:54.383138 21553 external_mini_cluster.cc:1428] Setting key b1228a8bd846da37ee908f780a999b22
I20260926 16:43:54.383167 537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.65:45471 every 8 connection(s)
WARNING: no policy specified for kudu/127.21.12.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.66@KRBTEST.COM" created.
Sep 26 16:43:54 dist-test-slave-n1hw krb5kdc[32316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441034, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:43:54.412151 542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:54.389730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:59039 (local address 127.21.12.126:39655)
0926 16:43:54.397195 (+ 7465us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:54.397200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:54.397216 (+ 16us) server_negotiation.cc:407] Connection header received
0926 16:43:54.397259 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:54.397263 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:54.397310 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:54.397400 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:54.399119 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:54.400041 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:54.403246 (+ 3205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:54.403595 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:54.403685 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:54.406774 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:54.406778 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:54.406781 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:54.406809 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:54.408758 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:54.409379 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:54.409382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:54.409384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:54.409476 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:54.409932 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:54.409936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:54.409938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:54.410159 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:54.410263 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:54.411477 (+ 1214us) server_negotiation.cc:299] Negotiation successful
0926 16:43:54.411606 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7384,"thread_start_us":102,"threads_started":1}
I20260926 16:43:54.412261 541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:54.388665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:39655 (local address 127.21.12.65:59039)
0926 16:43:54.394079 (+ 5414us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:54.394124 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:54.394895 (+ 771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:54.397725 (+ 2830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:54.397737 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:54.398237 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:54.398943 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:54.398964 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:54.400191 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:54.400195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:54.403086 (+ 2891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:54.403098 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:54.403860 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:54.403865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:54.404058 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:54.404772 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:54.404794 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:54.406555 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:54.408897 (+ 2342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:54.408903 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:54.408919 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:54.409258 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:54.409585 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:54.409589 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:54.409591 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:54.409805 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:54.410286 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:54.410293 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:54.410606 (+ 313us) client_negotiation.cc:769] Sending connection context
0926 16:43:54.410813 (+ 207us) client_negotiation.cc:240] Negotiation successful
0926 16:43:54.411173 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5200,"thread_start_us":135,"threads_started":1}
I20260926 16:43:54.413795 538 heartbeater.cc:347] Connected to a master server at 127.21.12.126:39655
I20260926 16:43:54.414059 538 heartbeater.cc:464] Registering TS with master...
I20260926 16:43:54.414654 538 heartbeater.cc:511] Master 127.21.12.126:39655 requested a full tablet report, sending...
I20260926 16:43:54.416394 324 ts_manager.cc:194] Registered new tserver with Master: 2cdbd3b405fc4dd3b1a2502c7aad9df8 (127.21.12.65:45471)
I20260926 16:43:54.417996 324 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.65@KRBTEST.COM'} at 127.21.12.65:59039
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab.
Entry for principal kudu/127.21.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab.
I20260926 16:43:54.422484 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.12.84:55169/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.66:0
--local_ip_for_outbound_sockets=127.21.12.66
--webserver_interface=127.21.12.66
--webserver_port=0
--tserver_master_addrs=127.21.12.126:39655
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab
--principal=kudu/127.21.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:43945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 16:43:54 dist-test-slave-n1hw krb5kdc[32316](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790441034, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 16:43:54.577099 544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:54.577436 544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:54.577502 544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:54.583582 544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:54.583716 544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:54.583838 544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.66
I20260926 16:43:54.591532 544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:43945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.21.12.84:55169/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab
--principal=kudu/127.21.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.66
--webserver_port=0
--tserver_master_addrs=127.21.12.126:39655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.66
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:43:54.593027 544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:54.594388 544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:54.606050 550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:54.606024 553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:54.606330 551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:54.607179 544 server_base.cc:1034] running on GCE node
I20260926 16:43:54.607599 544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:54.608350 544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:54.609552 544 hybrid_clock.cc:648] HybridClock initialized: now 1790441034609521 us; error 44 us; skew 500 ppm
Sep 26 16:43:54 dist-test-slave-n1hw krb5kdc[32316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441034, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:43:54.613083 544 init.cc:377] Logged in from keytab as kudu/127.21.12.66@KRBTEST.COM (short username kudu)
I20260926 16:43:54.614099 544 webserver.cc:460] Webserver started at http://127.21.12.66:33757/ using document root <none> and password file <none>
I20260926 16:43:54.614763 544 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:54.614846 544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:54.615064 544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 16:43:54 dist-test-slave-n1hw krb5kdc[32316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441034, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.66@KRBTEST.COM for HTTP/127.21.12.84@KRBTEST.COM
I20260926 16:43:54.631469 544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-1/data/instance:
uuid: "b79a7386e7d3487cbbf349708c8e0e9a"
format_stamp: "Formatted at 2026-09-26 16:43:54 on dist-test-slave-n1hw"
server_key: "b28c59b03362ff595364609717dfc021"
server_key_iv: "046c615e5923dcb39c5f3ddc0ddf2251"
server_key_version: "kuduclusterkey@0"
I20260926 16:43:54.632267 544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance:
uuid: "b79a7386e7d3487cbbf349708c8e0e9a"
format_stamp: "Formatted at 2026-09-26 16:43:54 on dist-test-slave-n1hw"
server_key: "b28c59b03362ff595364609717dfc021"
server_key_iv: "046c615e5923dcb39c5f3ddc0ddf2251"
server_key_version: "kuduclusterkey@0"
I20260926 16:43:54.650257 544 fs_manager.cc:691] Time spent creating directory manager: real 0.018s user 0.004s sys 0.000s
I20260926 16:43:54.673599 560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:54.675293 544 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.000s
I20260926 16:43:54.675419 544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-1/wal
uuid: "b79a7386e7d3487cbbf349708c8e0e9a"
format_stamp: "Formatted at 2026-09-26 16:43:54 on dist-test-slave-n1hw"
server_key: "b28c59b03362ff595364609717dfc021"
server_key_iv: "046c615e5923dcb39c5f3ddc0ddf2251"
server_key_version: "kuduclusterkey@0"
I20260926 16:43:54.675549 544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:54.699579 544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:54.703238 544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:54.703418 544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:54.704077 544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:43:54.705075 544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:43:54.705134 544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:54.705183 544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:43:54.705214 544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:54.717377 544 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.66:32919
I20260926 16:43:54.717527 673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.66:32919 every 8 connection(s)
I20260926 16:43:54.718859 544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
I20260926 16:43:54.722040 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 544
I20260926 16:43:54.722165 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance
I20260926 16:43:54.722465 21553 external_mini_cluster.cc:1428] Setting key 98a6739a1948d573794e4abd3df5ea0b
WARNING: no policy specified for kudu/127.21.12.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.67@KRBTEST.COM" created.
Sep 26 16:43:54 dist-test-slave-n1hw krb5kdc[32316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441034, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.66@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:43:54.757249 542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:54.720992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:44849 (local address 127.21.12.126:39655)
0926 16:43:54.721126 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:54.721130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:54.723470 (+ 2340us) server_negotiation.cc:407] Connection header received
0926 16:43:54.726754 (+ 3284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:54.726758 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:54.726816 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:54.726877 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:54.728520 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:54.729336 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:54.731132 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:54.731482 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:54.731559 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:54.752886 (+ 21327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:54.752891 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:54.752894 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:54.752927 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:54.754747 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:54.755382 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:54.755386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:54.755388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:54.755474 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:54.755890 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:54.755893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:54.755895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:54.756136 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:54.756227 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:54.756943 (+ 716us) server_negotiation.cc:299] Negotiation successful
0926 16:43:54.757084 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 16:43:54.759151 676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:54.721468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:39655 (local address 127.21.12.66:44849)
0926 16:43:54.723317 (+ 1849us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:54.723373 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:54.726472 (+ 3099us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:54.727271 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:54.727283 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:54.727644 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:54.728344 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:54.728366 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:54.729484 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:54.729488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:54.730993 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:54.731004 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:54.749745 (+ 18741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:54.749771 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:54.750125 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:54.750928 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:54.750951 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:54.752663 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:54.754885 (+ 2222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:54.754892 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:54.754910 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:54.755250 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:54.755585 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:54.755588 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:54.755590 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:54.755777 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:54.756297 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:54.756304 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:54.756638 (+ 334us) client_negotiation.cc:769] Sending connection context
0926 16:43:54.757698 (+ 1060us) client_negotiation.cc:240] Negotiation successful
0926 16:43:54.758079 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1623,"thread_start_us":293,"threads_started":1}
I20260926 16:43:54.760689 674 heartbeater.cc:347] Connected to a master server at 127.21.12.126:39655
I20260926 16:43:54.760926 674 heartbeater.cc:464] Registering TS with master...
I20260926 16:43:54.761478 674 heartbeater.cc:511] Master 127.21.12.126:39655 requested a full tablet report, sending...
I20260926 16:43:54.762651 324 ts_manager.cc:194] Registered new tserver with Master: b79a7386e7d3487cbbf349708c8e0e9a (127.21.12.66:32919)
I20260926 16:43:54.763406 324 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.66@KRBTEST.COM'} at 127.21.12.66:44849
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab.
Entry for principal kudu/127.21.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab.
I20260926 16:43:54.781081 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.12.84:55169/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.67:0
--local_ip_for_outbound_sockets=127.21.12.67
--webserver_interface=127.21.12.67
--webserver_port=0
--tserver_master_addrs=127.21.12.126:39655
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab
--principal=kudu/127.21.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:43945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 16:43:54 dist-test-slave-n1hw krb5kdc[32316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441034, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for HTTP/127.21.12.84@KRBTEST.COM
W20260926 16:43:54.941627 679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:54.942063 679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:54.942167 679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:54.947786 679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:54.947903 679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:54.948022 679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.67
I20260926 16:43:54.955456 679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:43945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.21.12.84:55169/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab
--principal=kudu/127.21.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.67
--webserver_port=0
--tserver_master_addrs=127.21.12.126:39655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.67
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:43:54.956913 679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:54.958230 679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:54.976008 687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:54.976409 685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:54.976632 684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:54.984112 679 server_base.cc:1034] running on GCE node
I20260926 16:43:54.984705 679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:54.985561 679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:54.993008 679 hybrid_clock.cc:648] HybridClock initialized: now 1790441034992925 us; error 95 us; skew 500 ppm
Sep 26 16:43:54 dist-test-slave-n1hw krb5kdc[32316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441034, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:43:54.996851 679 init.cc:377] Logged in from keytab as kudu/127.21.12.67@KRBTEST.COM (short username kudu)
I20260926 16:43:54.998131 679 webserver.cc:460] Webserver started at http://127.21.12.67:43037/ using document root <none> and password file <none>
I20260926 16:43:54.998888 679 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:54.999027 679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:54.999302 679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 16:43:54 dist-test-slave-n1hw krb5kdc[32316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441034, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.67@KRBTEST.COM for HTTP/127.21.12.84@KRBTEST.COM
I20260926 16:43:55.015919 679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-2/data/instance:
uuid: "0c48cc19a8ce468890448851b0fcbf8e"
format_stamp: "Formatted at 2026-09-26 16:43:55 on dist-test-slave-n1hw"
server_key: "cc1a82fbc3d8ddb1a142852024edc058"
server_key_iv: "ec9c6e1d71ba736299eaa1b8e0987a8b"
server_key_version: "kuduclusterkey@0"
I20260926 16:43:55.016700 679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance:
uuid: "0c48cc19a8ce468890448851b0fcbf8e"
format_stamp: "Formatted at 2026-09-26 16:43:55 on dist-test-slave-n1hw"
server_key: "cc1a82fbc3d8ddb1a142852024edc058"
server_key_iv: "ec9c6e1d71ba736299eaa1b8e0987a8b"
server_key_version: "kuduclusterkey@0"
I20260926 16:43:55.021005 679 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:43:55.045006 694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:55.046427 679 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 16:43:55.046566 679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-2/wal
uuid: "0c48cc19a8ce468890448851b0fcbf8e"
format_stamp: "Formatted at 2026-09-26 16:43:55 on dist-test-slave-n1hw"
server_key: "cc1a82fbc3d8ddb1a142852024edc058"
server_key_iv: "ec9c6e1d71ba736299eaa1b8e0987a8b"
server_key_version: "kuduclusterkey@0"
I20260926 16:43:55.046710 679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:55.062510 679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:55.065704 679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:55.065933 679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:55.066561 679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:43:55.067400 679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:43:55.067471 679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:55.067565 679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:43:55.067616 679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:55.075657 679 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.67:45355
I20260926 16:43:55.076539 679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
I20260926 16:43:55.079748 807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.67:45355 every 8 connection(s)
I20260926 16:43:55.084749 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 679
I20260926 16:43:55.084885 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance
I20260926 16:43:55.085217 21553 external_mini_cluster.cc:1428] Setting key e630a8d1e9f2f79b8b68af0a0ec7ea72
Sep 26 16:43:55 dist-test-slave-n1hw krb5kdc[32316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441034, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.67@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:43:55.097533 542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:55.083650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:42341 (local address 127.21.12.126:39655)
0926 16:43:55.083792 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:55.083797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:55.084425 (+ 628us) server_negotiation.cc:407] Connection header received
0926 16:43:55.085629 (+ 1204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:55.085632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:55.085688 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:55.085779 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:55.087359 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:55.088153 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:55.089807 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:55.090164 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:55.090246 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:55.093299 (+ 3053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:55.093303 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:55.093306 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:55.093332 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:55.095159 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:55.095827 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:55.095831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:55.095833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:55.095918 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:55.096302 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:55.096305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:55.096307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:55.096518 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:55.096589 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:55.097248 (+ 659us) server_negotiation.cc:299] Negotiation successful
0926 16:43:55.097372 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 16:43:55.099150 810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:55.082496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:39655 (local address 127.21.12.67:42341)
0926 16:43:55.084263 (+ 1767us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:55.084317 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:55.085377 (+ 1060us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:55.086186 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:55.086195 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:55.086544 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:55.087199 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:55.087217 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:55.088310 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:55.088312 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:55.089627 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:55.089638 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:55.090481 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:55.090485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:55.090683 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:55.091425 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:55.091446 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:55.093114 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:55.095349 (+ 2235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:55.095355 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:55.095370 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:55.095709 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:55.096020 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:55.096023 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:55.096025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:55.096189 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:55.096667 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:55.096673 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:55.096970 (+ 297us) client_negotiation.cc:769] Sending connection context
0926 16:43:55.097767 (+ 797us) client_negotiation.cc:240] Negotiation successful
0926 16:43:55.098081 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1543,"thread_start_us":160,"threads_started":1}
I20260926 16:43:55.100531 808 heartbeater.cc:347] Connected to a master server at 127.21.12.126:39655
I20260926 16:43:55.100760 808 heartbeater.cc:464] Registering TS with master...
I20260926 16:43:55.101266 808 heartbeater.cc:511] Master 127.21.12.126:39655 requested a full tablet report, sending...
I20260926 16:43:55.102337 324 ts_manager.cc:194] Registered new tserver with Master: 0c48cc19a8ce468890448851b0fcbf8e (127.21.12.67:45355)
I20260926 16:43:55.103060 324 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.67@KRBTEST.COM'} at 127.21.12.67:42341
I20260926 16:43:55.111148 21553 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:43:55.122064 542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:55.113627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51434 (local address 127.21.12.126:39655)
0926 16:43:55.113799 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:55.113803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:55.114092 (+ 289us) server_negotiation.cc:407] Connection header received
0926 16:43:55.114255 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:55.114259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:55.114308 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:55.114390 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:55.115109 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:55.115622 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:55.116485 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:55.116720 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:55.116825 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:55.118337 (+ 1512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:55.118341 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:55.118345 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:55.118376 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:55.120121 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:55.120639 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:55.120643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:55.120645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:55.120741 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:55.121078 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:55.121081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:55.121083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:55.121307 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:55.121418 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:55.121803 (+ 385us) server_negotiation.cc:299] Negotiation successful
0926 16:43:55.121916 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 16:43:55.132818 324 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51434:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:43:55.135970 324 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:43:55.151953 821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:55.144949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:32919 (local address 127.0.0.1:42422)
0926 16:43:55.146283 (+ 1334us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:55.146332 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:55.146461 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:55.147230 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:55.147234 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:55.147252 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:43:55.147353 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:55.147358 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:55.148373 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:55.148376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:55.149743 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:55.149789 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:55.151410 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:55.151413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:55.151553 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:55.151586 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 16:43:55.151679 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 16:43:55.151798 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1146,"mutex_wait_us":29,"thread_start_us":87,"threads_started":1}
I20260926 16:43:55.152395 824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:55.145723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42422 (local address 127.21.12.66:32919)
0926 16:43:55.146744 (+ 1021us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:55.146756 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:55.146779 (+ 23us) server_negotiation.cc:407] Connection header received
0926 16:43:55.146855 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:55.146860 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:55.147067 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:55.147243 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:43:55.147468 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:55.148251 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:55.149997 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:55.151023 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:55.151101 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:55.151322 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:55.152121 (+ 799us) server_negotiation.cc:299] Negotiation successful
0926 16:43:55.152239 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":917,"thread_start_us":128,"threads_started":1}
I20260926 16:43:55.153947 823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:55.145019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:45471 (local address 127.0.0.1:38260)
0926 16:43:55.146771 (+ 1752us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:55.146784 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:55.146924 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:55.148242 (+ 1318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:55.148245 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:55.148264 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:43:55.148370 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:55.148375 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:55.150585 (+ 2210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:55.150589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:55.151732 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:55.151740 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:55.153559 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:55.153562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:55.153646 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:55.153667 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 16:43:55.153747 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 16:43:55.153829 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1684,"thread_start_us":88,"threads_started":1}
I20260926 16:43:55.154382 822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:55.145681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38260 (local address 127.21.12.65:45471)
0926 16:43:55.147804 (+ 2123us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:55.147810 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:55.147832 (+ 22us) server_negotiation.cc:407] Connection header received
0926 16:43:55.147904 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:55.147909 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:55.148097 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:55.148260 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:43:55.148491 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:55.150460 (+ 1969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:55.151851 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:55.153278 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:55.153376 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:55.153499 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:55.154166 (+ 667us) server_negotiation.cc:299] Negotiation successful
0926 16:43:55.154256 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2023,"thread_start_us":143,"threads_started":1}
I20260926 16:43:55.155498 596 tablet_service.cc:1467] Processing CreateTablet for tablet fc1df3c0b6894018af89def947aab2fb (DEFAULT_TABLE table=test-table [id=d210639438c448c998c0bdf363d023f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:43:55.156527 596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc1df3c0b6894018af89def947aab2fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:55.156993 472 tablet_service.cc:1467] Processing CreateTablet for tablet fc1df3c0b6894018af89def947aab2fb (DEFAULT_TABLE table=test-table [id=d210639438c448c998c0bdf363d023f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:43:55.158021 472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc1df3c0b6894018af89def947aab2fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:55.158450 820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:55.145801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.67:45355 (local address 127.0.0.1:60496)
0926 16:43:55.147480 (+ 1679us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:55.147494 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:55.147576 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:55.148986 (+ 1410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:55.148990 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:55.149008 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:43:55.149089 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:55.149095 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:55.150735 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:55.150739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:55.151965 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:55.151977 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:55.158081 (+ 6104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:55.158085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:55.158170 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:55.158193 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 16:43:55.158244 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 16:43:55.158317 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1618,"mutex_wait_us":1,"thread_start_us":78,"threads_started":1}
I20260926 16:43:55.159265 825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:55.146210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60496 (local address 127.21.12.67:45355)
0926 16:43:55.148543 (+ 2333us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:55.148554 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:55.148575 (+ 21us) server_negotiation.cc:407] Connection header received
0926 16:43:55.148645 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:55.148650 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:55.148827 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:55.149297 (+ 470us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:43:55.149437 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:55.150603 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:55.152393 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:55.154270 (+ 1877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:55.158845 (+ 4575us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:55.158983 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:55.159031 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 16:43:55.159125 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2231,"thread_start_us":116,"threads_started":1}
I20260926 16:43:55.161988 741 tablet_service.cc:1467] Processing CreateTablet for tablet fc1df3c0b6894018af89def947aab2fb (DEFAULT_TABLE table=test-table [id=d210639438c448c998c0bdf363d023f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:43:55.162935 741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc1df3c0b6894018af89def947aab2fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:55.164911 827 tablet_bootstrap.cc:492] T fc1df3c0b6894018af89def947aab2fb P 2cdbd3b405fc4dd3b1a2502c7aad9df8: Bootstrap starting.
I20260926 16:43:55.167224 827 tablet_bootstrap.cc:654] T fc1df3c0b6894018af89def947aab2fb P 2cdbd3b405fc4dd3b1a2502c7aad9df8: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:55.168062 827 log.cc:824] T fc1df3c0b6894018af89def947aab2fb P 2cdbd3b405fc4dd3b1a2502c7aad9df8: Log is configured to *not* fsync() on all Append() calls
I20260926 16:43:55.169502 829 tablet_bootstrap.cc:492] T fc1df3c0b6894018af89def947aab2fb P 0c48cc19a8ce468890448851b0fcbf8e: Bootstrap starting.
I20260926 16:43:55.170192 827 tablet_bootstrap.cc:492] T fc1df3c0b6894018af89def947aab2fb P 2cdbd3b405fc4dd3b1a2502c7aad9df8: No bootstrap required, opened a new log
I20260926 16:43:55.170378 827 ts_tablet_manager.cc:1397] T fc1df3c0b6894018af89def947aab2fb P 2cdbd3b405fc4dd3b1a2502c7aad9df8: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 16:43:55.172345 829 tablet_bootstrap.cc:654] T fc1df3c0b6894018af89def947aab2fb P 0c48cc19a8ce468890448851b0fcbf8e: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:55.172986 829 log.cc:824] T fc1df3c0b6894018af89def947aab2fb P 0c48cc19a8ce468890448851b0fcbf8e: Log is configured to *not* fsync() on all Append() calls
I20260926 16:43:55.174077 826 tablet_bootstrap.cc:492] T fc1df3c0b6894018af89def947aab2fb P b79a7386e7d3487cbbf349708c8e0e9a: Bootstrap starting.
I20260926 16:43:55.174832 829 tablet_bootstrap.cc:492] T fc1df3c0b6894018af89def947aab2fb P 0c48cc19a8ce468890448851b0fcbf8e: No bootstrap required, opened a new log
I20260926 16:43:55.174813 827 raft_consensus.cc:348] T fc1df3c0b6894018af89def947aab2fb P 2cdbd3b405fc4dd3b1a2502c7aad9df8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b79a7386e7d3487cbbf349708c8e0e9a" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 32919 } } peers { permanent_uuid: "2cdbd3b405fc4dd3b1a2502c7aad9df8" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45471 } } peers { permanent_uuid: "0c48cc19a8ce468890448851b0fcbf8e" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 45355 } }
I20260926 16:43:55.175025 829 ts_tablet_manager.cc:1397] T fc1df3c0b6894018af89def947aab2fb P 0c48cc19a8ce468890448851b0fcbf8e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 16:43:55.175056 827 raft_consensus.cc:374] T fc1df3c0b6894018af89def947aab2fb P 2cdbd3b405fc4dd3b1a2502c7aad9df8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:55.175108 827 raft_consensus.cc:729] T fc1df3c0b6894018af89def947aab2fb P 2cdbd3b405fc4dd3b1a2502c7aad9df8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cdbd3b405fc4dd3b1a2502c7aad9df8, State: Initialized, Role: FOLLOWER
I20260926 16:43:55.175563 827 consensus_queue.cc:260] T fc1df3c0b6894018af89def947aab2fb P 2cdbd3b405fc4dd3b1a2502c7aad9df8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b79a7386e7d3487cbbf349708c8e0e9a" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 32919 } } peers { permanent_uuid: "2cdbd3b405fc4dd3b1a2502c7aad9df8" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45471 } } peers { permanent_uuid: "0c48cc19a8ce468890448851b0fcbf8e" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 45355 } }
I20260926 16:43:55.176544 827 ts_tablet_manager.cc:1428] T fc1df3c0b6894018af89def947aab2fb P 2cdbd3b405fc4dd3b1a2502c7aad9df8: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 16:43:55.176896 538 heartbeater.cc:503] Master 127.21.12.126:39655 was elected leader, sending a full tablet report...
I20260926 16:43:55.177049 826 tablet_bootstrap.cc:654] T fc1df3c0b6894018af89def947aab2fb P b79a7386e7d3487cbbf349708c8e0e9a: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:55.177897 826 log.cc:824] T fc1df3c0b6894018af89def947aab2fb P b79a7386e7d3487cbbf349708c8e0e9a: Log is configured to *not* fsync() on all Append() calls
I20260926 16:43:55.178954 829 raft_consensus.cc:348] T fc1df3c0b6894018af89def947aab2fb P 0c48cc19a8ce468890448851b0fcbf8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b79a7386e7d3487cbbf349708c8e0e9a" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 32919 } } peers { permanent_uuid: "2cdbd3b405fc4dd3b1a2502c7aad9df8" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45471 } } peers { permanent_uuid: "0c48cc19a8ce468890448851b0fcbf8e" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 45355 } }
I20260926 16:43:55.179237 829 raft_consensus.cc:374] T fc1df3c0b6894018af89def947aab2fb P 0c48cc19a8ce468890448851b0fcbf8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:55.179298 829 raft_consensus.cc:729] T fc1df3c0b6894018af89def947aab2fb P 0c48cc19a8ce468890448851b0fcbf8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c48cc19a8ce468890448851b0fcbf8e, State: Initialized, Role: FOLLOWER
I20260926 16:43:55.179723 829 consensus_queue.cc:260] T fc1df3c0b6894018af89def947aab2fb P 0c48cc19a8ce468890448851b0fcbf8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b79a7386e7d3487cbbf349708c8e0e9a" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 32919 } } peers { permanent_uuid: "2cdbd3b405fc4dd3b1a2502c7aad9df8" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45471 } } peers { permanent_uuid: "0c48cc19a8ce468890448851b0fcbf8e" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 45355 } }
I20260926 16:43:55.180629 829 ts_tablet_manager.cc:1428] T fc1df3c0b6894018af89def947aab2fb P 0c48cc19a8ce468890448851b0fcbf8e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 16:43:55.180946 808 heartbeater.cc:503] Master 127.21.12.126:39655 was elected leader, sending a full tablet report...
I20260926 16:43:55.182711 826 tablet_bootstrap.cc:492] T fc1df3c0b6894018af89def947aab2fb P b79a7386e7d3487cbbf349708c8e0e9a: No bootstrap required, opened a new log
I20260926 16:43:55.182901 826 ts_tablet_manager.cc:1397] T fc1df3c0b6894018af89def947aab2fb P b79a7386e7d3487cbbf349708c8e0e9a: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.002s
I20260926 16:43:55.186271 826 raft_consensus.cc:348] T fc1df3c0b6894018af89def947aab2fb P b79a7386e7d3487cbbf349708c8e0e9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b79a7386e7d3487cbbf349708c8e0e9a" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 32919 } } peers { permanent_uuid: "2cdbd3b405fc4dd3b1a2502c7aad9df8" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45471 } } peers { permanent_uuid: "0c48cc19a8ce468890448851b0fcbf8e" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 45355 } }
I20260926 16:43:55.186480 826 raft_consensus.cc:374] T fc1df3c0b6894018af89def947aab2fb P b79a7386e7d3487cbbf349708c8e0e9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:55.186539 826 raft_consensus.cc:729] T fc1df3c0b6894018af89def947aab2fb P b79a7386e7d3487cbbf349708c8e0e9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b79a7386e7d3487cbbf349708c8e0e9a, State: Initialized, Role: FOLLOWER
I20260926 16:43:55.186952 826 consensus_queue.cc:260] T fc1df3c0b6894018af89def947aab2fb P b79a7386e7d3487cbbf349708c8e0e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b79a7386e7d3487cbbf349708c8e0e9a" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 32919 } } peers { permanent_uuid: "2cdbd3b405fc4dd3b1a2502c7aad9df8" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45471 } } peers { permanent_uuid: "0c48cc19a8ce468890448851b0fcbf8e" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 45355 } }
I20260926 16:43:55.187824 826 ts_tablet_manager.cc:1428] T fc1df3c0b6894018af89def947aab2fb P b79a7386e7d3487cbbf349708c8e0e9a: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260926 16:43:55.188161 674 heartbeater.cc:503] Master 127.21.12.126:39655 was elected leader, sending a full tablet report...
W20260926 16:43:55.227594 675 tablet.cc:2369] T fc1df3c0b6894018af89def947aab2fb P b79a7386e7d3487cbbf349708c8e0e9a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 16:43:55 dist-test-slave-n1hw krb5kdc[32316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441034, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for HTTP/127.21.12.84@KRBTEST.COM
I20260926 16:43:55.276007 831 raft_consensus.cc:482] T fc1df3c0b6894018af89def947aab2fb P 2cdbd3b405fc4dd3b1a2502c7aad9df8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:43:55.276230 831 raft_consensus.cc:504] T fc1df3c0b6894018af89def947aab2fb P 2cdbd3b405fc4dd3b1a2502c7aad9df8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b79a7386e7d3487cbbf349708c8e0e9a" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 32919 } } peers { permanent_uuid: "2cdbd3b405fc4dd3b1a2502c7aad9df8" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45471 } } peers { permanent_uuid: "0c48cc19a8ce468890448851b0fcbf8e" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 45355 } }
I20260926 16:43:55.277599 831 leader_election.cc:288] T fc1df3c0b6894018af89def947aab2fb P 2cdbd3b405fc4dd3b1a2502c7aad9df8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b79a7386e7d3487cbbf349708c8e0e9a (127.21.12.66:32919), 0c48cc19a8ce468890448851b0fcbf8e (127.21.12.67:45355)
I20260926 16:43:55.285483 825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:55.278083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:34245 (local address 127.21.12.67:45355)
0926 16:43:55.278242 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:55.278249 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:55.278540 (+ 291us) server_negotiation.cc:407] Connection header received
0926 16:43:55.279734 (+ 1194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:55.279737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:55.279787 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:55.279835 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:43:55.280452 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:55.281133 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:55.284028 (+ 2895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:55.284501 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:55.284575 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:55.284614 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:55.285276 (+ 662us) server_negotiation.cc:299] Negotiation successful
0926 16:43:55.285354 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 16:43:55.286253 837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:55.277996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.67:45355 (local address 127.21.12.65:34245)
0926 16:43:55.278435 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:55.278451 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:55.278571 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:55.280148 (+ 1577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:55.280151 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:55.280172 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:43:55.280281 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:55.280287 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:55.281601 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:55.281604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:55.283727 (+ 2123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:55.283738 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:55.284854 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:55.284857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:55.284963 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:55.284988 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 16:43:55.286008 (+ 1020us) client_negotiation.cc:240] Negotiation successful
0926 16:43:55.286085 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":124,"threads_started":1}
I20260926 16:43:55.286953 762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1df3c0b6894018af89def947aab2fb" candidate_uuid: "2cdbd3b405fc4dd3b1a2502c7aad9df8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c48cc19a8ce468890448851b0fcbf8e" is_pre_election: true
I20260926 16:43:55.287173 836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:55.278144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:32919 (local address 127.21.12.65:47073)
0926 16:43:55.278761 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:55.278774 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:55.278865 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:55.279122 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:55.279125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:55.279144 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:43:55.279289 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:55.279296 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:55.284273 (+ 4977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:55.284277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:55.285512 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:55.285523 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:55.286760 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:55.286764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:55.286857 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:55.286877 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 16:43:55.286964 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 16:43:55.287031 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":104,"threads_started":1}
I20260926 16:43:55.287226 824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:55.277944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:47073 (local address 127.21.12.66:32919)
0926 16:43:55.278080 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:55.278084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:55.278856 (+ 772us) server_negotiation.cc:407] Connection header received
0926 16:43:55.278954 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:55.278957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:55.279010 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:55.279095 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:43:55.279412 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:55.280240 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:55.285675 (+ 5435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:55.286474 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:55.286642 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:55.286693 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:55.287001 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 16:43:55.287098 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 16:43:55.287345 762 raft_consensus.cc:2454] T fc1df3c0b6894018af89def947aab2fb P 0c48cc19a8ce468890448851b0fcbf8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cdbd3b405fc4dd3b1a2502c7aad9df8 in term 0.
I20260926 16:43:55.287758 610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1df3c0b6894018af89def947aab2fb" candidate_uuid: "2cdbd3b405fc4dd3b1a2502c7aad9df8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b79a7386e7d3487cbbf349708c8e0e9a" is_pre_election: true
I20260926 16:43:55.287868 423 leader_election.cc:302] T fc1df3c0b6894018af89def947aab2fb P 2cdbd3b405fc4dd3b1a2502c7aad9df8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c48cc19a8ce468890448851b0fcbf8e, 2cdbd3b405fc4dd3b1a2502c7aad9df8; no voters:
I20260926 16:43:55.288043 610 raft_consensus.cc:2454] T fc1df3c0b6894018af89def947aab2fb P b79a7386e7d3487cbbf349708c8e0e9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cdbd3b405fc4dd3b1a2502c7aad9df8 in term 0.
I20260926 16:43:55.288152 831 raft_consensus.cc:2781] T fc1df3c0b6894018af89def947aab2fb P 2cdbd3b405fc4dd3b1a2502c7aad9df8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:43:55.288231 831 raft_consensus.cc:482] T fc1df3c0b6894018af89def947aab2fb P 2cdbd3b405fc4dd3b1a2502c7aad9df8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:43:55.288280 831 raft_consensus.cc:3037] T fc1df3c0b6894018af89def947aab2fb P 2cdbd3b405fc4dd3b1a2502c7aad9df8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:55.289340 831 raft_consensus.cc:504] T fc1df3c0b6894018af89def947aab2fb P 2cdbd3b405fc4dd3b1a2502c7aad9df8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b79a7386e7d3487cbbf349708c8e0e9a" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 32919 } } peers { permanent_uuid: "2cdbd3b405fc4dd3b1a2502c7aad9df8" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45471 } } peers { permanent_uuid: "0c48cc19a8ce468890448851b0fcbf8e" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 45355 } }
I20260926 16:43:55.289713 831 leader_election.cc:288] T fc1df3c0b6894018af89def947aab2fb P 2cdbd3b405fc4dd3b1a2502c7aad9df8 [CANDIDATE]: Term 1 election: Requested vote from peers b79a7386e7d3487cbbf349708c8e0e9a (127.21.12.66:32919), 0c48cc19a8ce468890448851b0fcbf8e (127.21.12.67:45355)
I20260926 16:43:55.290020 610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1df3c0b6894018af89def947aab2fb" candidate_uuid: "2cdbd3b405fc4dd3b1a2502c7aad9df8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b79a7386e7d3487cbbf349708c8e0e9a"
I20260926 16:43:55.290107 762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1df3c0b6894018af89def947aab2fb" candidate_uuid: "2cdbd3b405fc4dd3b1a2502c7aad9df8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c48cc19a8ce468890448851b0fcbf8e"
I20260926 16:43:55.290170 610 raft_consensus.cc:3037] T fc1df3c0b6894018af89def947aab2fb P b79a7386e7d3487cbbf349708c8e0e9a [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:55.290237 762 raft_consensus.cc:3037] T fc1df3c0b6894018af89def947aab2fb P 0c48cc19a8ce468890448851b0fcbf8e [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:55.291272 610 raft_consensus.cc:2454] T fc1df3c0b6894018af89def947aab2fb P b79a7386e7d3487cbbf349708c8e0e9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cdbd3b405fc4dd3b1a2502c7aad9df8 in term 1.
I20260926 16:43:55.291293 762 raft_consensus.cc:2454] T fc1df3c0b6894018af89def947aab2fb P 0c48cc19a8ce468890448851b0fcbf8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cdbd3b405fc4dd3b1a2502c7aad9df8 in term 1.
I20260926 16:43:55.291565 426 leader_election.cc:302] T fc1df3c0b6894018af89def947aab2fb P 2cdbd3b405fc4dd3b1a2502c7aad9df8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cdbd3b405fc4dd3b1a2502c7aad9df8, b79a7386e7d3487cbbf349708c8e0e9a; no voters:
I20260926 16:43:55.291719 831 raft_consensus.cc:2781] T fc1df3c0b6894018af89def947aab2fb P 2cdbd3b405fc4dd3b1a2502c7aad9df8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:43:55.291956 831 raft_consensus.cc:686] T fc1df3c0b6894018af89def947aab2fb P 2cdbd3b405fc4dd3b1a2502c7aad9df8 [term 1 LEADER]: Becoming Leader. State: Replica: 2cdbd3b405fc4dd3b1a2502c7aad9df8, State: Running, Role: LEADER
I20260926 16:43:55.292279 831 consensus_queue.cc:237] T fc1df3c0b6894018af89def947aab2fb P 2cdbd3b405fc4dd3b1a2502c7aad9df8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b79a7386e7d3487cbbf349708c8e0e9a" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 32919 } } peers { permanent_uuid: "2cdbd3b405fc4dd3b1a2502c7aad9df8" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45471 } } peers { permanent_uuid: "0c48cc19a8ce468890448851b0fcbf8e" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 45355 } }
I20260926 16:43:55.299057 323 catalog_manager.cc:5495] T fc1df3c0b6894018af89def947aab2fb P 2cdbd3b405fc4dd3b1a2502c7aad9df8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2cdbd3b405fc4dd3b1a2502c7aad9df8 (127.21.12.65). New cstate: current_term: 1 leader_uuid: "2cdbd3b405fc4dd3b1a2502c7aad9df8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b79a7386e7d3487cbbf349708c8e0e9a" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 32919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cdbd3b405fc4dd3b1a2502c7aad9df8" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 45471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c48cc19a8ce468890448851b0fcbf8e" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 45355 } health_report { overall_health: UNKNOWN } } }
W20260926 16:43:55.330932 809 tablet.cc:2369] T fc1df3c0b6894018af89def947aab2fb P 0c48cc19a8ce468890448851b0fcbf8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:43:55.346011 822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:55.340784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38274 (local address 127.21.12.65:45471)
0926 16:43:55.340906 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:55.340911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:55.340924 (+ 13us) server_negotiation.cc:407] Connection header received
0926 16:43:55.341034 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:55.341038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:55.341118 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:55.341224 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:43:55.341736 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:55.342603 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:55.343838 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:55.344172 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:55.344274 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:55.344692 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:43:55.345258 (+ 566us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:43:55.345381 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:55.345808 (+ 427us) server_negotiation.cc:299] Negotiation successful
0926 16:43:55.345871 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 16:43:55.353942 762 raft_consensus.cc:1261] T fc1df3c0b6894018af89def947aab2fb P 0c48cc19a8ce468890448851b0fcbf8e [term 1 FOLLOWER]: Refusing update from remote peer 2cdbd3b405fc4dd3b1a2502c7aad9df8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:43:55.353972 610 raft_consensus.cc:1261] T fc1df3c0b6894018af89def947aab2fb P b79a7386e7d3487cbbf349708c8e0e9a [term 1 FOLLOWER]: Refusing update from remote peer 2cdbd3b405fc4dd3b1a2502c7aad9df8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:43:55.354718 831 consensus_queue.cc:1035] T fc1df3c0b6894018af89def947aab2fb P 2cdbd3b405fc4dd3b1a2502c7aad9df8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b79a7386e7d3487cbbf349708c8e0e9a" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 32919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:43:55.354930 838 consensus_queue.cc:1035] T fc1df3c0b6894018af89def947aab2fb P 2cdbd3b405fc4dd3b1a2502c7aad9df8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c48cc19a8ce468890448851b0fcbf8e" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:43:55.373138 842 mvcc.cc:205] Tried to move back new op lower bound from 7333646480801701888 to 7333646480560271360. Current Snapshot: MvccSnapshot[applied={T|T < 7333646480801701888}]
I20260926 16:43:55.378332 843 mvcc.cc:205] Tried to move back new op lower bound from 7333646480801701888 to 7333646480560271360. Current Snapshot: MvccSnapshot[applied={T|T < 7333646480801701888}]
I20260926 16:43:55.391508 324 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:43:55.391685 324 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:43:55.394259 324 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8e7fa2bbcd49447ab16d1389884a2d42: Sending DeleteTablet for 3 replicas of tablet fc1df3c0b6894018af89def947aab2fb
I20260926 16:43:55.395265 472 tablet_service.cc:1514] Processing DeleteTablet for tablet fc1df3c0b6894018af89def947aab2fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:43:55 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:38260
I20260926 16:43:55.395411 741 tablet_service.cc:1514] Processing DeleteTablet for tablet fc1df3c0b6894018af89def947aab2fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:43:55 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:60496
I20260926 16:43:55.396128 596 tablet_service.cc:1514] Processing DeleteTablet for tablet fc1df3c0b6894018af89def947aab2fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:43:55 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:42422
I20260926 16:43:55.396988 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 405
I20260926 16:43:55.400462 853 tablet_replica.cc:330] T fc1df3c0b6894018af89def947aab2fb P b79a7386e7d3487cbbf349708c8e0e9a: stopping tablet replica
I20260926 16:43:55.400807 853 raft_consensus.cc:2229] T fc1df3c0b6894018af89def947aab2fb P b79a7386e7d3487cbbf349708c8e0e9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:43:55.401211 853 raft_consensus.cc:2258] T fc1df3c0b6894018af89def947aab2fb P b79a7386e7d3487cbbf349708c8e0e9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:43:55.402729 853 ts_tablet_manager.cc:1905] T fc1df3c0b6894018af89def947aab2fb P b79a7386e7d3487cbbf349708c8e0e9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:43:55.402995 852 tablet_replica.cc:330] T fc1df3c0b6894018af89def947aab2fb P 0c48cc19a8ce468890448851b0fcbf8e: stopping tablet replica
I20260926 16:43:55.403317 852 raft_consensus.cc:2229] T fc1df3c0b6894018af89def947aab2fb P 0c48cc19a8ce468890448851b0fcbf8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:43:55.403699 852 raft_consensus.cc:2258] T fc1df3c0b6894018af89def947aab2fb P 0c48cc19a8ce468890448851b0fcbf8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:43:55.406635 853 ts_tablet_manager.cc:1918] T fc1df3c0b6894018af89def947aab2fb P b79a7386e7d3487cbbf349708c8e0e9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:43:55.407899 852 ts_tablet_manager.cc:1905] T fc1df3c0b6894018af89def947aab2fb P 0c48cc19a8ce468890448851b0fcbf8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:43:55.410547 852 ts_tablet_manager.cc:1918] T fc1df3c0b6894018af89def947aab2fb P 0c48cc19a8ce468890448851b0fcbf8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:43:55.410719 853 log.cc:1196] T fc1df3c0b6894018af89def947aab2fb P b79a7386e7d3487cbbf349708c8e0e9a: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-1/wal/wals/fc1df3c0b6894018af89def947aab2fb
I20260926 16:43:55.411100 853 ts_tablet_manager.cc:1939] T fc1df3c0b6894018af89def947aab2fb P b79a7386e7d3487cbbf349708c8e0e9a: Deleting consensus metadata
I20260926 16:43:55.412417 311 catalog_manager.cc:4841] TS b79a7386e7d3487cbbf349708c8e0e9a (127.21.12.66:32919): tablet fc1df3c0b6894018af89def947aab2fb (table test-table [id=d210639438c448c998c0bdf363d023f8]) successfully deleted
I20260926 16:43:55.412700 852 log.cc:1196] T fc1df3c0b6894018af89def947aab2fb P 0c48cc19a8ce468890448851b0fcbf8e: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440924937351-21553-0/minicluster-data/ts-2/wal/wals/fc1df3c0b6894018af89def947aab2fb
I20260926 16:43:55.413113 852 ts_tablet_manager.cc:1939] T fc1df3c0b6894018af89def947aab2fb P 0c48cc19a8ce468890448851b0fcbf8e: Deleting consensus metadata
I20260926 16:43:55.414307 308 catalog_manager.cc:4841] TS 0c48cc19a8ce468890448851b0fcbf8e (127.21.12.67:45355): tablet fc1df3c0b6894018af89def947aab2fb (table test-table [id=d210639438c448c998c0bdf363d023f8]) successfully deleted
W20260926 16:43:55.415232 310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.12.65:45471 (error 108)
I20260926 16:43:55.416591 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 544
I20260926 16:43:55.416625 820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:55.416354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:45471 (local address 127.0.0.1:38284)
0926 16:43:55.416468 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:55.416540 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.65:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 16:43:55.417249 310 catalog_manager.cc:4584] TS 2cdbd3b405fc4dd3b1a2502c7aad9df8 (127.21.12.65:45471): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fc1df3c0b6894018af89def947aab2fb: Network error: Client connection negotiation failed: client connection to 127.21.12.65:45471: connect: Connection refused (error 111)
I20260926 16:43:55.424372 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 679
I20260926 16:43:55.431000 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 32760
I20260926 16:43:55.436606 21553 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 16:43:55.781234 21553 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 16:43:55.781355 21553 mini_ranger.cc:66] Stopping Ranger...
I20260926 16:43:56.126075 21553 mini_ranger.cc:68] Stopped Ranger
2026-09-26 16:43:56.126 UTC [32346] LOG: received smart shutdown request
2026-09-26 16:43:56.129 UTC [32346] LOG: background worker "logical replication launcher" (PID 32355) exited with exit code 1
2026-09-26 16:43:56.130 UTC [32350] LOG: shutting down
2026-09-26 16:43:56.306 UTC [32346] LOG: database system is shut down
2026-09-26T16:43:56Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46054 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:43:56 dist-test-slave-n1hw krb5kdc[863](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:43:56 dist-test-slave-n1hw krb5kdc[863](info): set up 2 sockets
Sep 26 16:43:56 dist-test-slave-n1hw krb5kdc[863](info): commencing operation
krb5kdc: starting...
W20260926 16:43:57.472045 21553 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Sep 26 16:43:57 dist-test-slave-n1hw krb5kdc[863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:43:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:43:57Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.21.12.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.21.12.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.21.12.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/rangeradmin_127.21.12.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.21.12.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/rangeradmin_127.21.12.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.21.12.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.21.12.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.21.12.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/rangerlookup_127.21.12.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.21.12.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/rangerlookup_127.21.12.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.12.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.21.12.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.12.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/HTTP_127.21.12.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.12.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/HTTP_127.21.12.84@KRBTEST.COM.keytab.
I20260926 16:43:57.636619 21553 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskEZzU7d/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/postgres -l logfile start
2026-09-26 16:44:00.184 UTC [893] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 16:44:00.184 UTC [893] LOG: listening on IPv4 address "127.21.12.84", port 47863
2026-09-26 16:44:00.188 UTC [893] LOG: listening on Unix socket "/tmp/.s.PGSQL.47863"
2026-09-26 16:44:00.209 UTC [896] LOG: database system was shut down at 2026-09-26 16:43:58 UTC
2026-09-26 16:44:00.213 UTC [893] LOG: database system is ready to accept connections
I20260926 16:44:01.231841 21553 mini_postgres.cc:96] Postgres bound to 47863
2026-09-26 16:44:01.238 UTC [904] FATAL: database "slave" does not exist
127.21.12.84:47863 - accepting connections
I20260926 16:44:01.238610 21553 mini_ranger.cc:161] Starting Ranger...
I20260926 16:44:01.253017 21553 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 16:44:01.757117 21553 mini_ranger.cc:87] Created ranger Postgres database
I20260926 16:44:01.757242 21553 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ranger-admin
2026-09-26 16:44:02,024 [I] DB FLAVOR :POSTGRES
2026-09-26 16:44:02,024 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 16:44:02,024 [I] Checking connection..
2026-09-26 16:44:02,024 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.84:47863/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 16:44:02,275 [I] Checking connection passed.
2026-09-26 16:44:02,275 [I] --------- Verifying version history table ---------
2026-09-26 16:44:02,276 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.84:47863/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 16:44:02,544 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 16:44:02,545 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.84:47863/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 16:44:02,803 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 16:44:02,803 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 16:44:02,803 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.84:47863/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 16:44:03.053 UTC [968] WARNING: there is no transaction in progress
2026-09-26 16:44:03,060 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 16:44:03,060 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.84:47863/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 16:44:03,304 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 16:44:03,305 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 16:44:03,305 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.84:47863/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 16:44:03,536 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.84:47863/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 16:44:03,791 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.84:47863/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-n1hw.c.gcp-upstream.internal','N') ;"
2026-09-26 16:44:04,028 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.84:47863/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 16:44:04,278 [I] Table x_portal_user does not exist in database ranger
2026-09-26 16:44:04,279 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.84:47863/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 16:44:04,534 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 16:44:04,535 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.84:47863/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 16:44:04,784 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.84:47863/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 16:44:05,006 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 16:44:05,006 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.84:47863/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 16:44:05.953 UTC [1076] WARNING: there is no transaction in progress
2026-09-26 16:44:05.969 UTC [1076] WARNING: there is no transaction in progress
2026-09-26 16:44:05.987 UTC [1076] WARNING: there is no transaction in progress
2026-09-26 16:44:06.004 UTC [1076] WARNING: there is no transaction in progress
2026-09-26 16:44:06.020 UTC [1076] WARNING: there is no transaction in progress
2026-09-26 16:44:06.040 UTC [1076] WARNING: there is no transaction in progress
2026-09-26 16:44:06.094 UTC [1076] WARNING: there is no transaction in progress
2026-09-26 16:44:06.100 UTC [1076] WARNING: there is no transaction in progress
2026-09-26 16:44:06.114 UTC [1076] WARNING: there is no transaction in progress
2026-09-26 16:44:06.124 UTC [1076] WARNING: there is no transaction in progress
2026-09-26 16:44:06.134 UTC [1076] WARNING: there is no transaction in progress
2026-09-26 16:44:06.147 UTC [1076] WARNING: there is no transaction in progress
2026-09-26 16:44:06.158 UTC [1076] WARNING: there is no transaction in progress
2026-09-26 16:44:06.164 UTC [1076] WARNING: there is no transaction in progress
2026-09-26 16:44:06.171 UTC [1076] WARNING: there is no transaction in progress
2026-09-26 16:44:06.178 UTC [1076] WARNING: there is no transaction in progress
2026-09-26 16:44:06,798 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 16:44:06,799 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.84:47863/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 16:44:07,033 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 16:44:07,034 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.84:47863/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 16:44:07,315 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 16:44:07,315 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.84:47863/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 16:44:07,554 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 16:44:07,554 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.84:47863/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 16:44:07,811 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.84:47863/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 16:44:08,059 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.84:47863/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-n1hw.c.gcp-upstream.internal';"
2026-09-26 16:44:08,297 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 16:44:08,298 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.84:47863/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 16:44:08,522 [I] DB_PATCHES have already been applied
I20260926 16:44:08.527765 21553 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ranger-admin:/tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskEZzU7d/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 16:44:08.527858 21553 mini_ranger.cc:193] Using host: 127.21.12.84
I20260926 16:44:08.531427 21553 mini_ranger.cc:227] Ranger admin URL: http://127.21.12.84:35409
Sep 26, 2026 4:44:08 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 4:44:08 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 4:44:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 4:44:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 4:44:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 4:44:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 4:44:09 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 16:44:09.635092 21553 mini_ranger.cc:160] Time spent starting Ranger: real 8.396s user 0.000s sys 0.009s
Sep 26, 2026 4:44:29 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 4:44:29 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 4:44:29 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 4:44:29 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 4:44:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 4:44:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 4:44:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 4:44:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 4:44:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 4:44:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 4:44:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 4:44:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 4:44:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 4:44:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 4:44:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 4:44:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 4:44:29 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 4:44:30 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 16:44:31.238442 21553 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.21.12.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.21.12.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.21.12.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/rangerkms_127.21.12.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.21.12.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/rangerkms_127.21.12.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.12.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.21.12.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.12.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/HTTP_127.21.12.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.12.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/HTTP_127.21.12.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 16:44:31 dist-test-slave-n1hw krb5kdc[863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441071, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 16:44:31.323426 21553 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 16:44:31.332656 21553 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 16:44:32.195791 21553 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 16:44:32.195894 21553 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ranger-kms
I20260926 16:44:32.195917 21553 mini_ranger_kms.cc:227] Using postgres at 127.21.12.84:47863
2026-09-26 16:44:32,375 [I] DB FLAVOR :POSTGRES
2026-09-26 16:44:32,375 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 16:44:32,375 [I] Checking connection
2026-09-26 16:44:32,376 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.84:47863/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 16:44:32,620 [I] connection success
2026-09-26 16:44:32,620 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 16:44:32,620 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 16:44:32,620 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.84:47863/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 16:44:32,846 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 16:44:32,846 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 16:44:32,846 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 16:44:32,846 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.84:47863/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 16:44:33,125 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 16:44:33.619079 21553 mini_ranger_kms.cc:310] Created kms service
I20260926 16:44:33.812124 21553 mini_ranger_kms.cc:326] Created kudu user
I20260926 16:44:33.854924 21553 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 16:44:33 dist-test-slave-n1hw krb5kdc[863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441071, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.21.12.84@KRBTEST.COM
I20260926 16:44:34.232456 21553 mini_ranger_kms.cc:384] Added ranger policy
I20260926 16:44:34.232605 21553 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ranger-kms:/tmp/dist-test-taskEZzU7d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskEZzU7d/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 16:44:34.232626 21553 mini_ranger_kms.cc:240] Using host: 127.21.12.84
I20260926 16:44:34.236274 21553 mini_ranger_kms.cc:276] Ranger KMS PID: 1256
I20260926 16:44:34.236356 21553 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.21.12.84:56695
Sep 26, 2026 4:44:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 4:44:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 4:44:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 4:44:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 4:44:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 4:44:35 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 26 16:44:36 dist-test-slave-n1hw krb5kdc[863](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790441076, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.12.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:36.486161 21553 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.163s user 0.007s sys 0.155s
I20260926 16:44:36.486294 21553 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 16:44:36.486374 21553 mini_ranger_kms.cc:401] 127.21.12.84:56695/kms/v1/keys
Sep 26 16:44:36 dist-test-slave-n1hw krb5kdc[863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441076, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.12.84@KRBTEST.COM for HTTP/127.21.12.84@KRBTEST.COM
Sep 26 16:44:37 dist-test-slave-n1hw krb5kdc[863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441076, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.12.84@KRBTEST.COM for HTTP/127.21.12.84@KRBTEST.COM
Sep 26 16:44:39 dist-test-slave-n1hw krb5kdc[863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.21.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
I20260926 16:44:39.477270 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:34869
--webserver_interface=127.21.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:41779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:34869
--ranger_config_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.12.84:56695/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:44:39.587647 1307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:39.587913 1307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:39.587950 1307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:39.592000 1307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:44:39.592058 1307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:39.592080 1307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:39.592096 1307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:44:39.592113 1307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:44:39.597440 1307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:41779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/master-0/wal
--ranger_kms_url=127.21.12.84:56695/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:34869
--ranger_config_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:34869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:44:39.598500 1307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:39.599532 1307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:39.608013 1312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:39.608024 1313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:39.608193 1307 server_base.cc:1034] running on GCE node
W20260926 16:44:39.608287 1315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:39.608765 1307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:39.609901 1307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:39.611071 1307 hybrid_clock.cc:648] HybridClock initialized: now 1790441079611051 us; error 43 us; skew 500 ppm
Sep 26 16:44:39 dist-test-slave-n1hw krb5kdc[863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441079, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:39.613927 1307 init.cc:377] Logged in from keytab as kudu/127.21.12.126@KRBTEST.COM (short username kudu)
I20260926 16:44:39.614916 1307 webserver.cc:460] Webserver started at http://127.21.12.126:34013/ using document root <none> and password file <none>
I20260926 16:44:39.615581 1307 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:39.615656 1307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:39.615863 1307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 16:44:39 dist-test-slave-n1hw krb5kdc[863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441079, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for HTTP/127.21.12.84@KRBTEST.COM
I20260926 16:44:39.661892 1307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "52665a9e0d8b43a09dcc7d0f18d87b8e"
format_stamp: "Formatted at 2026-09-26 16:44:39 on dist-test-slave-n1hw"
server_key: "45141e08b9f5cf6ff96ae5e31c20f790"
server_key_iv: "10bec383be5d76c75a261f0b33d7e7c5"
server_key_version: "kuduclusterkey@0"
I20260926 16:44:39.662554 1307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "52665a9e0d8b43a09dcc7d0f18d87b8e"
format_stamp: "Formatted at 2026-09-26 16:44:39 on dist-test-slave-n1hw"
server_key: "45141e08b9f5cf6ff96ae5e31c20f790"
server_key_iv: "10bec383be5d76c75a261f0b33d7e7c5"
server_key_version: "kuduclusterkey@0"
I20260926 16:44:39.671834 1307 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.004s sys 0.000s
I20260926 16:44:39.689181 1322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:39.690578 1307 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 16:44:39.690685 1307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "52665a9e0d8b43a09dcc7d0f18d87b8e"
format_stamp: "Formatted at 2026-09-26 16:44:39 on dist-test-slave-n1hw"
server_key: "45141e08b9f5cf6ff96ae5e31c20f790"
server_key_iv: "10bec383be5d76c75a261f0b33d7e7c5"
server_key_version: "kuduclusterkey@0"
I20260926 16:44:39.690831 1307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:39.701396 1307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:39.704282 1307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:39.704756 1307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:39.712196 1307 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.126:34869
I20260926 16:44:39.712250 1374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.126:34869 every 8 connection(s)
I20260926 16:44:39.713181 1307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:44:39.714212 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 1307
I20260926 16:44:39.714355 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:44:39.714641 21553 external_mini_cluster.cc:1428] Setting key 6f3e342293dfe545d340cfc9360addba
I20260926 16:44:39.717268 1375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 16:44:39 dist-test-slave-n1hw krb5kdc[863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:44:39.724104 1375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52665a9e0d8b43a09dcc7d0f18d87b8e: Bootstrap starting.
I20260926 16:44:39.726691 1375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52665a9e0d8b43a09dcc7d0f18d87b8e: Neither blocks nor log segments found. Creating new log.
I20260926 16:44:39.728040 1378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:39.716019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60278 (local address 127.21.12.126:34869)
0926 16:44:39.716409 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:39.716431 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:39.716464 (+ 33us) server_negotiation.cc:407] Connection header received
0926 16:44:39.717096 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:39.717112 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:39.717482 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:39.717834 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:39.718580 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:39.719564 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:39.720421 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:39.720956 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:39.721127 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:39.723593 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:39.723598 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:39.723610 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:39.723635 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:39.725252 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:39.725677 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:39.725682 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:39.725688 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:39.725869 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:39.726153 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:39.726157 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:39.726158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:39.726536 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:39.726749 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:39.727076 (+ 327us) server_negotiation.cc:299] Negotiation successful
0926 16:44:39.727353 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":136,"threads_started":1}
I20260926 16:44:39.728018 1375 log.cc:824] T 00000000000000000000000000000000 P 52665a9e0d8b43a09dcc7d0f18d87b8e: Log is configured to *not* fsync() on all Append() calls
I20260926 16:44:39.730144 1375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52665a9e0d8b43a09dcc7d0f18d87b8e: No bootstrap required, opened a new log
I20260926 16:44:39.733013 1375 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52665a9e0d8b43a09dcc7d0f18d87b8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52665a9e0d8b43a09dcc7d0f18d87b8e" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34869 } }
I20260926 16:44:39.733233 1375 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52665a9e0d8b43a09dcc7d0f18d87b8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:44:39.733305 1375 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52665a9e0d8b43a09dcc7d0f18d87b8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52665a9e0d8b43a09dcc7d0f18d87b8e, State: Initialized, Role: FOLLOWER
I20260926 16:44:39.733731 1375 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52665a9e0d8b43a09dcc7d0f18d87b8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52665a9e0d8b43a09dcc7d0f18d87b8e" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34869 } }
I20260926 16:44:39.733916 1375 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52665a9e0d8b43a09dcc7d0f18d87b8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:44:39.733978 1375 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52665a9e0d8b43a09dcc7d0f18d87b8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:44:39.734081 1375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52665a9e0d8b43a09dcc7d0f18d87b8e [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:44:39.734997 1375 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52665a9e0d8b43a09dcc7d0f18d87b8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52665a9e0d8b43a09dcc7d0f18d87b8e" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34869 } }
I20260926 16:44:39.735356 1375 leader_election.cc:302] T 00000000000000000000000000000000 P 52665a9e0d8b43a09dcc7d0f18d87b8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52665a9e0d8b43a09dcc7d0f18d87b8e; no voters:
I20260926 16:44:39.735684 1375 leader_election.cc:288] T 00000000000000000000000000000000 P 52665a9e0d8b43a09dcc7d0f18d87b8e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:44:39.735772 1380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52665a9e0d8b43a09dcc7d0f18d87b8e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:44:39.735973 1380 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52665a9e0d8b43a09dcc7d0f18d87b8e [term 1 LEADER]: Becoming Leader. State: Replica: 52665a9e0d8b43a09dcc7d0f18d87b8e, State: Running, Role: LEADER
I20260926 16:44:39.736307 1380 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52665a9e0d8b43a09dcc7d0f18d87b8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52665a9e0d8b43a09dcc7d0f18d87b8e" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34869 } }
I20260926 16:44:39.736836 1375 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52665a9e0d8b43a09dcc7d0f18d87b8e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:44:39.737511 1380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52665a9e0d8b43a09dcc7d0f18d87b8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52665a9e0d8b43a09dcc7d0f18d87b8e. Latest consensus state: current_term: 1 leader_uuid: "52665a9e0d8b43a09dcc7d0f18d87b8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52665a9e0d8b43a09dcc7d0f18d87b8e" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34869 } } }
I20260926 16:44:39.737970 1381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52665a9e0d8b43a09dcc7d0f18d87b8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52665a9e0d8b43a09dcc7d0f18d87b8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52665a9e0d8b43a09dcc7d0f18d87b8e" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34869 } } }
I20260926 16:44:39.738492 1381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52665a9e0d8b43a09dcc7d0f18d87b8e [sys.catalog]: This master's current role is: LEADER
I20260926 16:44:39.738384 1380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52665a9e0d8b43a09dcc7d0f18d87b8e [sys.catalog]: This master's current role is: LEADER
I20260926 16:44:39.738857 1393 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:44:39.739094 1375 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 16:44:39.742012 1393 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:44:39.753973 1393 catalog_manager.cc:1347] Generated new cluster ID: 4635deee851f459cb8b9aaed2bda14cb
I20260926 16:44:39.754050 1393 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:44:39.776120 1393 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:44:39.777520 1393 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:44:39.807054 1393 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52665a9e0d8b43a09dcc7d0f18d87b8e: Generated new TSK 0
I20260926 16:44:39.807999 1393 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab.
Entry for principal kudu/127.21.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab.
I20260926 16:44:40.495210 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.12.84:56695/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.65:0
--local_ip_for_outbound_sockets=127.21.12.65
--webserver_interface=127.21.12.65
--webserver_port=0
--tserver_master_addrs=127.21.12.126:34869
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab
--principal=kudu/127.21.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:41779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:44:40.638324 1420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:40.638564 1420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:40.638624 1420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:40.643227 1420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:40.643285 1420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:40.643366 1420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.65
I20260926 16:44:40.648748 1420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:41779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.21.12.84:56695/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab
--principal=kudu/127.21.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.65
--webserver_port=0
--tserver_master_addrs=127.21.12.126:34869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.65
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:44:40.649930 1420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:40.650972 1420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:40.660552 1428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:40.660560 1426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:40.660560 1425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:40.661141 1420 server_base.cc:1034] running on GCE node
I20260926 16:44:40.661566 1420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:40.662263 1420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:40.663439 1420 hybrid_clock.cc:648] HybridClock initialized: now 1790441080663410 us; error 41 us; skew 500 ppm
Sep 26 16:44:40 dist-test-slave-n1hw krb5kdc[863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441080, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:40.667111 1420 init.cc:377] Logged in from keytab as kudu/127.21.12.65@KRBTEST.COM (short username kudu)
I20260926 16:44:40.668231 1420 webserver.cc:460] Webserver started at http://127.21.12.65:42673/ using document root <none> and password file <none>
I20260926 16:44:40.668853 1420 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:40.668999 1420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:40.669288 1420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 16:44:40 dist-test-slave-n1hw krb5kdc[863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441080, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for HTTP/127.21.12.84@KRBTEST.COM
I20260926 16:44:40.684723 1420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-0/data/instance:
uuid: "57c8043747704341923c5c568ce64598"
format_stamp: "Formatted at 2026-09-26 16:44:40 on dist-test-slave-n1hw"
server_key: "7a388f439b75e666b069631a2d7c4ace"
server_key_iv: "244c2cb2c938432fa8d9a463635a2c8b"
server_key_version: "kuduclusterkey@0"
I20260926 16:44:40.685343 1420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance:
uuid: "57c8043747704341923c5c568ce64598"
format_stamp: "Formatted at 2026-09-26 16:44:40 on dist-test-slave-n1hw"
server_key: "7a388f439b75e666b069631a2d7c4ace"
server_key_iv: "244c2cb2c938432fa8d9a463635a2c8b"
server_key_version: "kuduclusterkey@0"
I20260926 16:44:40.689340 1420 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 16:44:40.718212 1437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:40.720043 1420 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.002s sys 0.000s
I20260926 16:44:40.720191 1420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-0/wal
uuid: "57c8043747704341923c5c568ce64598"
format_stamp: "Formatted at 2026-09-26 16:44:40 on dist-test-slave-n1hw"
server_key: "7a388f439b75e666b069631a2d7c4ace"
server_key_iv: "244c2cb2c938432fa8d9a463635a2c8b"
server_key_version: "kuduclusterkey@0"
I20260926 16:44:40.720322 1420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:40.789247 1420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:40.793141 1420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:40.793397 1420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:40.794332 1420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:44:40.795517 1420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:44:40.795626 1420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:40.795753 1420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:44:40.795832 1420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:40.805933 1420 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.65:36275
I20260926 16:44:40.806628 1552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.65:36275 every 8 connection(s)
I20260926 16:44:40.807221 1420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
I20260926 16:44:40.814647 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 1420
I20260926 16:44:40.814764 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance
I20260926 16:44:40.815059 21553 external_mini_cluster.cc:1428] Setting key 5012a569b15fcc4c9a434930075660e4
Sep 26 16:44:40 dist-test-slave-n1hw krb5kdc[863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441080, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:44:40.825524 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:40.810149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34869 (local address 127.21.12.65:36279)
0926 16:44:40.811281 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:40.811333 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:40.812333 (+ 1000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:40.812936 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:40.812946 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:40.813288 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:40.813951 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:40.813970 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:40.815170 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:40.815173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:40.816561 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:40.816572 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:40.817344 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:40.817352 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:40.817547 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:40.818220 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:40.818240 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:40.819970 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:40.822314 (+ 2344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:40.822321 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:40.822338 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:40.822670 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:40.822991 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:40.822994 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:40.822996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:40.823163 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:40.823619 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:40.823626 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:40.823936 (+ 310us) client_negotiation.cc:769] Sending connection context
0926 16:44:40.824124 (+ 188us) client_negotiation.cc:240] Negotiation successful
0926 16:44:40.824444 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":912,"thread_start_us":130,"threads_started":1}
I20260926 16:44:40.826323 1555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:40.809638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:36279 (local address 127.21.12.126:34869)
0926 16:44:40.809961 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:40.809967 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:40.811454 (+ 1487us) server_negotiation.cc:407] Connection header received
0926 16:44:40.812569 (+ 1115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:40.812574 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:40.812630 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:40.812741 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:40.814120 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:40.814999 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:40.816701 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:40.817059 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:40.817169 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:40.820172 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:40.820176 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:40.820179 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:40.820204 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:40.822181 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:40.822789 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:40.822793 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:40.822795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:40.822895 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:40.823262 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:40.823266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:40.823267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:40.823480 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:40.823568 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:40.826005 (+ 2437us) server_negotiation.cc:299] Negotiation successful
0926 16:44:40.826142 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":102,"threads_started":1}
WARNING: no policy specified for kudu/127.21.12.66@KRBTEST.COM; defaulting to no policy
I20260926 16:44:40.827586 1553 heartbeater.cc:347] Connected to a master server at 127.21.12.126:34869
I20260926 16:44:40.827836 1553 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:40.828433 1553 heartbeater.cc:511] Master 127.21.12.126:34869 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.66@KRBTEST.COM" created.
I20260926 16:44:40.830310 1339 ts_manager.cc:194] Registered new tserver with Master: 57c8043747704341923c5c568ce64598 (127.21.12.65:36275)
I20260926 16:44:40.831825 1339 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.65@KRBTEST.COM'} at 127.21.12.65:36279
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab.
Entry for principal kudu/127.21.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab.
I20260926 16:44:40.849143 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.12.84:56695/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.66:0
--local_ip_for_outbound_sockets=127.21.12.66
--webserver_interface=127.21.12.66
--webserver_port=0
--tserver_master_addrs=127.21.12.126:34869
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab
--principal=kudu/127.21.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:41779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 16:44:40 dist-test-slave-n1hw krb5kdc[863](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790441080, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 16:44:40.999547 1559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:40.999912 1559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:40.999980 1559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:41.006099 1559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:41.006208 1559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:41.006314 1559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.66
I20260926 16:44:41.014122 1559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:41779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.21.12.84:56695/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab
--principal=kudu/127.21.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.66
--webserver_port=0
--tserver_master_addrs=127.21.12.126:34869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.66
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:44:41.015435 1559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:41.016676 1559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:41.037305 1566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:41.037631 1568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:41.037910 1565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:41.039072 1559 server_base.cc:1034] running on GCE node
I20260926 16:44:41.039665 1559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:41.040443 1559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:41.041622 1559 hybrid_clock.cc:648] HybridClock initialized: now 1790441081041597 us; error 40 us; skew 500 ppm
Sep 26 16:44:41 dist-test-slave-n1hw krb5kdc[863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441081, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:41.049921 1559 init.cc:377] Logged in from keytab as kudu/127.21.12.66@KRBTEST.COM (short username kudu)
I20260926 16:44:41.051180 1559 webserver.cc:460] Webserver started at http://127.21.12.66:36629/ using document root <none> and password file <none>
I20260926 16:44:41.051920 1559 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:41.052001 1559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:41.052241 1559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 16:44:41 dist-test-slave-n1hw krb5kdc[863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441081, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.66@KRBTEST.COM for HTTP/127.21.12.84@KRBTEST.COM
I20260926 16:44:41.073273 1559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-1/data/instance:
uuid: "26cfed3ad4e64a6ebc4ba248d6230505"
format_stamp: "Formatted at 2026-09-26 16:44:41 on dist-test-slave-n1hw"
server_key: "38d83544d485a4ee1002d6b8795b17a1"
server_key_iv: "268b36154927a9efe8bf1f03f743d9db"
server_key_version: "kuduclusterkey@0"
I20260926 16:44:41.074067 1559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance:
uuid: "26cfed3ad4e64a6ebc4ba248d6230505"
format_stamp: "Formatted at 2026-09-26 16:44:41 on dist-test-slave-n1hw"
server_key: "38d83544d485a4ee1002d6b8795b17a1"
server_key_iv: "268b36154927a9efe8bf1f03f743d9db"
server_key_version: "kuduclusterkey@0"
I20260926 16:44:41.079187 1559 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 16:44:41.097522 1575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:41.103358 1559 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.000s
I20260926 16:44:41.103581 1559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-1/wal
uuid: "26cfed3ad4e64a6ebc4ba248d6230505"
format_stamp: "Formatted at 2026-09-26 16:44:41 on dist-test-slave-n1hw"
server_key: "38d83544d485a4ee1002d6b8795b17a1"
server_key_iv: "268b36154927a9efe8bf1f03f743d9db"
server_key_version: "kuduclusterkey@0"
I20260926 16:44:41.103788 1559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:41.125432 1559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:41.129321 1559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:41.129597 1559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:41.130419 1559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:44:41.131528 1559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:44:41.131640 1559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:41.131740 1559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:44:41.131809 1559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:41.150581 1559 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.66:44209
I20260926 16:44:41.152184 1559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
I20260926 16:44:41.154188 1688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.66:44209 every 8 connection(s)
I20260926 16:44:41.156759 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 1559
I20260926 16:44:41.156888 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance
I20260926 16:44:41.157228 21553 external_mini_cluster.cc:1428] Setting key 12f21f6efeaf8ec43a28fc9253713d8b
Sep 26 16:44:41 dist-test-slave-n1hw krb5kdc[863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441081, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.66@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:44:41.173106 1555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:41.158833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:51531 (local address 127.21.12.126:34869)
0926 16:44:41.158997 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:41.159002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:41.159422 (+ 420us) server_negotiation.cc:407] Connection header received
0926 16:44:41.160735 (+ 1313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:41.160739 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:41.160790 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:41.160848 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:41.162539 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:41.163364 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:41.165015 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:41.165360 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:41.165450 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:41.168706 (+ 3256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:41.168711 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:41.168713 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:41.168740 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:41.170613 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:41.171250 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:41.171254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:41.171255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:41.171343 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:41.171754 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:41.171757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:41.171759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:41.171986 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:41.172073 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:41.172774 (+ 701us) server_negotiation.cc:299] Negotiation successful
0926 16:44:41.172909 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 16:44:41.174758 1691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:41.157799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34869 (local address 127.21.12.66:51531)
0926 16:44:41.159276 (+ 1477us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:41.159332 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:41.160484 (+ 1152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:41.161248 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:41.161257 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:41.161642 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:41.162356 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:41.162377 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:41.163534 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:41.163537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:41.164877 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:41.164888 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:41.165650 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:41.165654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:41.165897 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:41.166771 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:41.166792 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:41.168510 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:41.170748 (+ 2238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:41.170754 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:41.170771 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:41.171114 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:41.171469 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:41.171472 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:41.171474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:41.171643 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:41.172141 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:41.172147 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:41.172468 (+ 321us) client_negotiation.cc:769] Sending connection context
0926 16:44:41.173334 (+ 866us) client_negotiation.cc:240] Negotiation successful
0926 16:44:41.173681 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1258,"thread_start_us":134,"threads_started":1}
I20260926 16:44:41.176317 1689 heartbeater.cc:347] Connected to a master server at 127.21.12.126:34869
I20260926 16:44:41.176579 1689 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:41.177129 1689 heartbeater.cc:511] Master 127.21.12.126:34869 requested a full tablet report, sending...
I20260926 16:44:41.178325 1339 ts_manager.cc:194] Registered new tserver with Master: 26cfed3ad4e64a6ebc4ba248d6230505 (127.21.12.66:44209)
I20260926 16:44:41.179044 1339 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.66@KRBTEST.COM'} at 127.21.12.66:51531
WARNING: no policy specified for kudu/127.21.12.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab.
Entry for principal kudu/127.21.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab.
I20260926 16:44:41.220742 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.12.84:56695/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.67:0
--local_ip_for_outbound_sockets=127.21.12.67
--webserver_interface=127.21.12.67
--webserver_port=0
--tserver_master_addrs=127.21.12.126:34869
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab
--principal=kudu/127.21.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:41779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 16:44:41 dist-test-slave-n1hw krb5kdc[863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441080, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for HTTP/127.21.12.84@KRBTEST.COM
W20260926 16:44:41.448125 1694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:41.448966 1694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:41.449118 1694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:41.454805 1694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:41.454936 1694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:41.455075 1694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.67
I20260926 16:44:41.462621 1694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:41779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.21.12.84:56695/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab
--principal=kudu/127.21.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.67
--webserver_port=0
--tserver_master_addrs=127.21.12.126:34869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.67
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:44:41.463940 1694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:41.465238 1694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:41.478102 1702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:41.478565 1694 server_base.cc:1034] running on GCE node
W20260926 16:44:41.478747 1700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:41.478943 1699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:41.479398 1694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:41.480091 1694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:41.481282 1694 hybrid_clock.cc:648] HybridClock initialized: now 1790441081481218 us; error 73 us; skew 500 ppm
Sep 26 16:44:41 dist-test-slave-n1hw krb5kdc[863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441081, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:41.484912 1694 init.cc:377] Logged in from keytab as kudu/127.21.12.67@KRBTEST.COM (short username kudu)
I20260926 16:44:41.485920 1694 webserver.cc:460] Webserver started at http://127.21.12.67:43247/ using document root <none> and password file <none>
I20260926 16:44:41.486562 1694 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:41.486627 1694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:41.486826 1694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 16:44:41 dist-test-slave-n1hw krb5kdc[863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441081, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.67@KRBTEST.COM for HTTP/127.21.12.84@KRBTEST.COM
I20260926 16:44:41.508127 1694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-2/data/instance:
uuid: "31160480af3f4de1ba0536eda6895a5b"
format_stamp: "Formatted at 2026-09-26 16:44:41 on dist-test-slave-n1hw"
server_key: "ac66df5eccc681f1ade2702ea06c3cb1"
server_key_iv: "5386b03fbc9f2fec0990872d15c8bc8a"
server_key_version: "kuduclusterkey@0"
I20260926 16:44:41.509014 1694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance:
uuid: "31160480af3f4de1ba0536eda6895a5b"
format_stamp: "Formatted at 2026-09-26 16:44:41 on dist-test-slave-n1hw"
server_key: "ac66df5eccc681f1ade2702ea06c3cb1"
server_key_iv: "5386b03fbc9f2fec0990872d15c8bc8a"
server_key_version: "kuduclusterkey@0"
I20260926 16:44:41.513949 1694 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 16:44:41.542173 1709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:41.543942 1694 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260926 16:44:41.544087 1694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-2/wal
uuid: "31160480af3f4de1ba0536eda6895a5b"
format_stamp: "Formatted at 2026-09-26 16:44:41 on dist-test-slave-n1hw"
server_key: "ac66df5eccc681f1ade2702ea06c3cb1"
server_key_iv: "5386b03fbc9f2fec0990872d15c8bc8a"
server_key_version: "kuduclusterkey@0"
I20260926 16:44:41.544230 1694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:41.563136 1694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:41.566839 1694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:41.567132 1694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:41.567890 1694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:44:41.569099 1694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:44:41.569200 1694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:41.569300 1694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:44:41.569370 1694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:41.583495 1694 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.67:36321
I20260926 16:44:41.584808 1694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
I20260926 16:44:41.587811 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 1694
I20260926 16:44:41.587942 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance
I20260926 16:44:41.588233 21553 external_mini_cluster.cc:1428] Setting key 864cf574e6ecabdb87c85a048a46169b
I20260926 16:44:41.586261 1822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.67:36321 every 8 connection(s)
Sep 26 16:44:41 dist-test-slave-n1hw krb5kdc[863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441081, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.67@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:44:41.609437 1555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:41.589802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:50655 (local address 127.21.12.126:34869)
0926 16:44:41.590116 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:41.590134 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:41.592465 (+ 2331us) server_negotiation.cc:407] Connection header received
0926 16:44:41.593709 (+ 1244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:41.593712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:41.593802 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:41.593855 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:41.595715 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:41.596559 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:41.598292 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:41.598634 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:41.598714 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:41.602334 (+ 3620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:41.602338 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:41.602341 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:41.602368 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:41.605320 (+ 2952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:41.607555 (+ 2235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:41.607560 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:41.607562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:41.607645 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:41.608096 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:41.608099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:41.608101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:41.608320 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:41.608407 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:41.609152 (+ 745us) server_negotiation.cc:299] Negotiation successful
0926 16:44:41.609275 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225}
I20260926 16:44:41.611382 1825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:41.588536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34869 (local address 127.21.12.67:50655)
0926 16:44:41.592324 (+ 3788us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:41.592376 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:41.593464 (+ 1088us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:41.594316 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:41.594327 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:41.594789 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:41.595540 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:41.595561 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:41.596719 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:41.596722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:41.598148 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:41.598159 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:41.598915 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:41.598919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:41.599136 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:41.599792 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:41.599813 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:41.601552 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:41.605460 (+ 3908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:41.605466 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:41.605483 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:41.605870 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:41.607768 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:41.607771 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:41.607773 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:41.607964 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:41.608477 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:41.608484 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:41.608866 (+ 382us) client_negotiation.cc:769] Sending connection context
0926 16:44:41.609962 (+ 1096us) client_negotiation.cc:240] Negotiation successful
0926 16:44:41.610330 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3572,"thread_start_us":134,"threads_started":1}
I20260926 16:44:41.612814 1823 heartbeater.cc:347] Connected to a master server at 127.21.12.126:34869
I20260926 16:44:41.613058 1823 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:41.613621 1823 heartbeater.cc:511] Master 127.21.12.126:34869 requested a full tablet report, sending...
I20260926 16:44:41.615343 1339 ts_manager.cc:194] Registered new tserver with Master: 31160480af3f4de1ba0536eda6895a5b (127.21.12.67:36321)
I20260926 16:44:41.616112 1339 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.67@KRBTEST.COM'} at 127.21.12.67:50655
I20260926 16:44:41.617673 21553 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:44:41.618444 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 1420
I20260926 16:44:41.628093 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 1559
I20260926 16:44:41.634686 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 1694
I20260926 16:44:41.648896 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 1307
I20260926 16:44:41.661509 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:34869
--webserver_interface=127.21.12.126
--webserver_port=34013
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:41779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:34869
--ranger_config_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.12.84:56695/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.12.84:56695/kms with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:44:41.768055 1826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:41.768322 1826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:41.768392 1826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:41.772342 1826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:44:41.772393 1826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:41.772413 1826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:41.772428 1826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:44:41.772480 1826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:44:41.777647 1826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:41779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.12.84:56695/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:34869
--ranger_config_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:34869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.126
--webserver_port=34013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:44:41.778621 1826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:41.779574 1826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 16:44:41.787706 1826 server_base.cc:1034] running on GCE node
W20260926 16:44:41.787642 1832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:41.787673 1834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:41.787642 1831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:41.788393 1826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:41.789417 1826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:41.790592 1826 hybrid_clock.cc:648] HybridClock initialized: now 1790441081790574 us; error 30 us; skew 500 ppm
Sep 26 16:44:41 dist-test-slave-n1hw krb5kdc[863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441081, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:41.793205 1826 init.cc:377] Logged in from keytab as kudu/127.21.12.126@KRBTEST.COM (short username kudu)
I20260926 16:44:41.794086 1826 webserver.cc:460] Webserver started at http://127.21.12.126:34013/ using document root <none> and password file <none>
I20260926 16:44:41.794590 1826 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:41.794677 1826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 16:44:41 dist-test-slave-n1hw krb5kdc[863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441081, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for HTTP/127.21.12.84@KRBTEST.COM
I20260926 16:44:41.841518 1826 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 16:44:41.843758 1842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:41.844954 1826 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 16:44:41.845104 1826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "52665a9e0d8b43a09dcc7d0f18d87b8e"
format_stamp: "Formatted at 2026-09-26 16:44:39 on dist-test-slave-n1hw"
server_key: "45141e08b9f5cf6ff96ae5e31c20f790"
server_key_iv: "10bec383be5d76c75a261f0b33d7e7c5"
server_key_version: "kuduclusterkey@0"
I20260926 16:44:41.845682 1826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:41.858608 1826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:41.861049 1826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:41.861188 1826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:41.867928 1826 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.126:34869
I20260926 16:44:41.867945 1894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.126:34869 every 8 connection(s)
I20260926 16:44:41.868988 1826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:44:41.872960 1895 sys_catalog.cc:263] Verifying existing consensus state
I20260926 16:44:41.874534 1895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52665a9e0d8b43a09dcc7d0f18d87b8e: Bootstrap starting.
I20260926 16:44:41.878197 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 1826
I20260926 16:44:41.878710 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.12.84:56695/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.65:36275
--local_ip_for_outbound_sockets=127.21.12.65
--tserver_master_addrs=127.21.12.126:34869
--webserver_port=42673
--webserver_interface=127.21.12.65
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab
--principal=kudu/127.21.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:41779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.12.84:56695/kms with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:44:41.891549 1895 log.cc:824] T 00000000000000000000000000000000 P 52665a9e0d8b43a09dcc7d0f18d87b8e: Log is configured to *not* fsync() on all Append() calls
I20260926 16:44:41.898289 1895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52665a9e0d8b43a09dcc7d0f18d87b8e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 16:44:41.898684 1895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52665a9e0d8b43a09dcc7d0f18d87b8e: Bootstrap complete.
I20260926 16:44:41.902915 1895 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52665a9e0d8b43a09dcc7d0f18d87b8e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52665a9e0d8b43a09dcc7d0f18d87b8e" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34869 } }
I20260926 16:44:41.903307 1895 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52665a9e0d8b43a09dcc7d0f18d87b8e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52665a9e0d8b43a09dcc7d0f18d87b8e, State: Initialized, Role: FOLLOWER
I20260926 16:44:41.903792 1895 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52665a9e0d8b43a09dcc7d0f18d87b8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52665a9e0d8b43a09dcc7d0f18d87b8e" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34869 } }
I20260926 16:44:41.903899 1895 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52665a9e0d8b43a09dcc7d0f18d87b8e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:44:41.903982 1895 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52665a9e0d8b43a09dcc7d0f18d87b8e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:44:41.904083 1895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52665a9e0d8b43a09dcc7d0f18d87b8e [term 1 FOLLOWER]: Advancing to term 2
I20260926 16:44:41.905932 1895 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52665a9e0d8b43a09dcc7d0f18d87b8e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52665a9e0d8b43a09dcc7d0f18d87b8e" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34869 } }
I20260926 16:44:41.906251 1895 leader_election.cc:302] T 00000000000000000000000000000000 P 52665a9e0d8b43a09dcc7d0f18d87b8e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52665a9e0d8b43a09dcc7d0f18d87b8e; no voters:
I20260926 16:44:41.906569 1895 leader_election.cc:288] T 00000000000000000000000000000000 P 52665a9e0d8b43a09dcc7d0f18d87b8e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 16:44:41.906659 1899 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52665a9e0d8b43a09dcc7d0f18d87b8e [term 2 FOLLOWER]: Leader election won for term 2
I20260926 16:44:41.906848 1899 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52665a9e0d8b43a09dcc7d0f18d87b8e [term 2 LEADER]: Becoming Leader. State: Replica: 52665a9e0d8b43a09dcc7d0f18d87b8e, State: Running, Role: LEADER
I20260926 16:44:41.907128 1899 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52665a9e0d8b43a09dcc7d0f18d87b8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52665a9e0d8b43a09dcc7d0f18d87b8e" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34869 } }
I20260926 16:44:41.907748 1895 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52665a9e0d8b43a09dcc7d0f18d87b8e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:44:41.908025 1899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52665a9e0d8b43a09dcc7d0f18d87b8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52665a9e0d8b43a09dcc7d0f18d87b8e. Latest consensus state: current_term: 2 leader_uuid: "52665a9e0d8b43a09dcc7d0f18d87b8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52665a9e0d8b43a09dcc7d0f18d87b8e" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34869 } } }
I20260926 16:44:41.908145 1899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52665a9e0d8b43a09dcc7d0f18d87b8e [sys.catalog]: This master's current role is: LEADER
I20260926 16:44:41.908478 1903 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:44:41.908607 1900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52665a9e0d8b43a09dcc7d0f18d87b8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "52665a9e0d8b43a09dcc7d0f18d87b8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52665a9e0d8b43a09dcc7d0f18d87b8e" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34869 } } }
I20260926 16:44:41.908775 1900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52665a9e0d8b43a09dcc7d0f18d87b8e [sys.catalog]: This master's current role is: LEADER
I20260926 16:44:41.911646 1903 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:44:41.912132 1895 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 16:44:41.912331 1903 catalog_manager.cc:1259] Loaded cluster ID: 4635deee851f459cb8b9aaed2bda14cb
I20260926 16:44:41.912384 1903 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:44:41.919222 1903 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:44:41.920341 1903 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 52665a9e0d8b43a09dcc7d0f18d87b8e: Loaded TSK: 0
I20260926 16:44:41.921697 1903 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 16:44:41.998147 1897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:41.998474 1897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:41.998553 1897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:42.005561 1897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:42.005633 1897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:42.005733 1897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.65
I20260926 16:44:42.014107 1897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:41779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.12.84:56695/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab
--principal=kudu/127.21.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.65:36275
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.65
--webserver_port=42673
--tserver_master_addrs=127.21.12.126:34869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.65
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:44:42.015370 1897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:42.016646 1897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:42.028867 1932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:42.029282 1935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:42.029716 1897 server_base.cc:1034] running on GCE node
W20260926 16:44:42.029780 1933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:42.030206 1897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:42.030966 1897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:42.032181 1897 hybrid_clock.cc:648] HybridClock initialized: now 1790441082032149 us; error 43 us; skew 500 ppm
Sep 26 16:44:42 dist-test-slave-n1hw krb5kdc[863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441082, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:42.035773 1897 init.cc:377] Logged in from keytab as kudu/127.21.12.65@KRBTEST.COM (short username kudu)
I20260926 16:44:42.036834 1897 webserver.cc:460] Webserver started at http://127.21.12.65:42673/ using document root <none> and password file <none>
I20260926 16:44:42.037524 1897 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:42.037633 1897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 16:44:42 dist-test-slave-n1hw krb5kdc[863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441082, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for HTTP/127.21.12.84@KRBTEST.COM
I20260926 16:44:42.071252 1897 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:44:42.078482 1943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:42.080178 1897 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.000s sys 0.002s
I20260926 16:44:42.080304 1897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-0/wal
uuid: "57c8043747704341923c5c568ce64598"
format_stamp: "Formatted at 2026-09-26 16:44:40 on dist-test-slave-n1hw"
server_key: "7a388f439b75e666b069631a2d7c4ace"
server_key_iv: "244c2cb2c938432fa8d9a463635a2c8b"
server_key_version: "kuduclusterkey@0"
I20260926 16:44:42.081037 1897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:42.096542 1897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:42.099694 1897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:42.099884 1897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:42.100589 1897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:44:42.101462 1897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:44:42.101528 1897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:42.101599 1897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:44:42.101629 1897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:42.112468 1897 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.65:36275
I20260926 16:44:42.112500 2058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.65:36275 every 8 connection(s)
I20260926 16:44:42.113633 1897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
I20260926 16:44:42.120432 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 1897
I20260926 16:44:42.120934 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.12.84:56695/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.66:44209
--local_ip_for_outbound_sockets=127.21.12.66
--tserver_master_addrs=127.21.12.126:34869
--webserver_port=36629
--webserver_interface=127.21.12.66
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab
--principal=kudu/127.21.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:41779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.12.84:56695/kms with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 16:44:42 dist-test-slave-n1hw krb5kdc[863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441082, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:44:42.137336 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:42.117865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34869 (local address 127.21.12.65:38793)
0926 16:44:42.119365 (+ 1500us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:42.119423 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:42.120522 (+ 1099us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:42.122406 (+ 1884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:42.122415 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:42.122783 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:42.123400 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:42.123420 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:42.125174 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:42.125177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:42.126608 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:42.126619 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:42.127809 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:42.127816 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:42.128075 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:42.128831 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:42.128857 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:42.130748 (+ 1891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:42.133613 (+ 2865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:42.133620 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:42.133637 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:42.134070 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:42.134526 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:42.134531 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:42.134535 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:42.134715 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:42.135435 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:42.135443 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:42.135748 (+ 305us) client_negotiation.cc:769] Sending connection context
0926 16:44:42.135962 (+ 214us) client_negotiation.cc:240] Negotiation successful
0926 16:44:42.136311 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1273,"thread_start_us":166,"threads_started":1}
I20260926 16:44:42.137358 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:42.118419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:38793 (local address 127.21.12.126:34869)
0926 16:44:42.119032 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:42.119054 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:42.119569 (+ 515us) server_negotiation.cc:407] Connection header received
0926 16:44:42.121119 (+ 1550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:42.121134 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:42.121547 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:42.121939 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:42.123591 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:42.125022 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:42.126737 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:42.127605 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:42.129708 (+ 2103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:42.130981 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:42.130990 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:42.131023 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:42.131065 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:42.133460 (+ 2395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:42.134231 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:42.134238 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:42.134246 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:42.134400 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:42.134832 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:42.134835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:42.134836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:42.135279 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:42.135407 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:42.136116 (+ 709us) server_negotiation.cc:299] Negotiation successful
0926 16:44:42.136395 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":128,"threads_started":1}
I20260926 16:44:42.140591 2059 heartbeater.cc:347] Connected to a master server at 127.21.12.126:34869
I20260926 16:44:42.140810 2059 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:42.141350 2059 heartbeater.cc:511] Master 127.21.12.126:34869 requested a full tablet report, sending...
W20260926 16:44:42.143539 2059 heartbeater.cc:644] Failed to heartbeat to 127.21.12.126:34869 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 16:44:42.143651 2059 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:42.143798 2059 heartbeater.cc:511] Master 127.21.12.126:34869 requested a full tablet report, sending...
I20260926 16:44:42.144680 2059 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:42.144825 2059 heartbeater.cc:511] Master 127.21.12.126:34869 requested a full tablet report, sending...
W20260926 16:44:42.145530 2059 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 16:44:42.261183 2063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:42.261538 2063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:42.261619 2063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:42.267622 2063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:42.267700 2063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:42.267795 2063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.66
I20260926 16:44:42.275504 2063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:41779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.12.84:56695/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab
--principal=kudu/127.21.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.66:44209
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.66
--webserver_port=36629
--tserver_master_addrs=127.21.12.126:34869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.66
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:44:42.276607 2063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:42.277609 2063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:42.287580 2069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:42.287720 2068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:42.288066 2063 server_base.cc:1034] running on GCE node
W20260926 16:44:42.288942 2071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:42.289409 2063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:42.290218 2063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:42.291421 2063 hybrid_clock.cc:648] HybridClock initialized: now 1790441082291356 us; error 79 us; skew 500 ppm
Sep 26 16:44:42 dist-test-slave-n1hw krb5kdc[863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441082, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:42.294957 2063 init.cc:377] Logged in from keytab as kudu/127.21.12.66@KRBTEST.COM (short username kudu)
I20260926 16:44:42.296092 2063 webserver.cc:460] Webserver started at http://127.21.12.66:36629/ using document root <none> and password file <none>
I20260926 16:44:42.296833 2063 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:42.296947 2063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 16:44:42 dist-test-slave-n1hw krb5kdc[863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441082, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.66@KRBTEST.COM for HTTP/127.21.12.84@KRBTEST.COM
I20260926 16:44:42.329090 2063 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:44:42.332033 2079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:42.333364 2063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 16:44:42.333506 2063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-1/wal
uuid: "26cfed3ad4e64a6ebc4ba248d6230505"
format_stamp: "Formatted at 2026-09-26 16:44:41 on dist-test-slave-n1hw"
server_key: "38d83544d485a4ee1002d6b8795b17a1"
server_key_iv: "268b36154927a9efe8bf1f03f743d9db"
server_key_version: "kuduclusterkey@0"
I20260926 16:44:42.334146 2063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:42.358364 2063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:42.361578 2063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:42.361838 2063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:42.362428 2063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:44:42.363252 2063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:44:42.363318 2063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:42.363386 2063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:44:42.363435 2063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:42.379098 2063 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.66:44209
I20260926 16:44:42.380317 2063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
I20260926 16:44:42.380666 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 2063
I20260926 16:44:42.381179 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.12.84:56695/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.67:36321
--local_ip_for_outbound_sockets=127.21.12.67
--tserver_master_addrs=127.21.12.126:34869
--webserver_port=43247
--webserver_interface=127.21.12.67
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab
--principal=kudu/127.21.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:41779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.12.84:56695/kms with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:44:42.386255 2193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.66:44209 every 8 connection(s)
Sep 26 16:44:42 dist-test-slave-n1hw krb5kdc[863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441082, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.66@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:44:42.419837 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:42.389031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:40529 (local address 127.21.12.126:34869)
0926 16:44:42.389192 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:42.389196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:42.402281 (+ 13085us) server_negotiation.cc:407] Connection header received
0926 16:44:42.403579 (+ 1298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:42.403583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:42.403640 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:42.403694 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:42.405333 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:42.406191 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:42.407932 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:42.408302 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:42.408391 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:42.411645 (+ 3254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:42.411650 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:42.411652 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:42.411683 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:42.417573 (+ 5890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:42.418253 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:42.418258 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:42.418259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:42.418370 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:42.418763 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:42.418766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:42.418768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:42.418957 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:42.419060 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:42.419538 (+ 478us) server_negotiation.cc:299] Negotiation successful
0926 16:44:42.419668 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 16:44:42.420687 2197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:42.389523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34869 (local address 127.21.12.66:40529)
0926 16:44:42.402084 (+ 12561us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:42.402150 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:42.403293 (+ 1143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:42.404109 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:42.404119 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:42.404481 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:42.405162 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:42.405180 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:42.406354 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:42.406357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:42.407792 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:42.407803 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:42.408670 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:42.408675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:42.408882 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:42.409658 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:42.409680 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:42.411455 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:42.417714 (+ 6259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:42.417720 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:42.417734 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:42.418126 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:42.418505 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:42.418511 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:42.418514 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:42.418659 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:42.419071 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:42.419078 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:42.419305 (+ 227us) client_negotiation.cc:769] Sending connection context
0926 16:44:42.419477 (+ 172us) client_negotiation.cc:240] Negotiation successful
0926 16:44:42.419772 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12295,"thread_start_us":166,"threads_started":1}
I20260926 16:44:42.422016 2194 heartbeater.cc:347] Connected to a master server at 127.21.12.126:34869
I20260926 16:44:42.422260 2194 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:42.422853 2194 heartbeater.cc:511] Master 127.21.12.126:34869 requested a full tablet report, sending...
W20260926 16:44:42.425045 2194 heartbeater.cc:644] Failed to heartbeat to 127.21.12.126:34869 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 16:44:42.425172 2194 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:42.425344 2194 heartbeater.cc:511] Master 127.21.12.126:34869 requested a full tablet report, sending...
I20260926 16:44:42.426225 2194 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:42.426397 2194 heartbeater.cc:511] Master 127.21.12.126:34869 requested a full tablet report, sending...
W20260926 16:44:42.427205 2194 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 16:44:42.543538 2196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:42.543889 2196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:42.543963 2196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:42.550001 2196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:42.550073 2196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:42.550174 2196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.67
I20260926 16:44:42.557988 2196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:41779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.12.84:56695/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab
--principal=kudu/127.21.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.67:36321
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.67
--webserver_port=43247
--tserver_master_addrs=127.21.12.126:34869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.67
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:44:42.559329 2196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:42.560673 2196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:42.572775 2204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:42.572793 2203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:42.573011 2196 server_base.cc:1034] running on GCE node
W20260926 16:44:42.573510 2206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:42.574016 2196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:42.574792 2196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:42.575987 2196 hybrid_clock.cc:648] HybridClock initialized: now 1790441082575922 us; error 82 us; skew 500 ppm
Sep 26 16:44:42 dist-test-slave-n1hw krb5kdc[863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441082, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:42.579366 2196 init.cc:377] Logged in from keytab as kudu/127.21.12.67@KRBTEST.COM (short username kudu)
I20260926 16:44:42.580493 2196 webserver.cc:460] Webserver started at http://127.21.12.67:43247/ using document root <none> and password file <none>
I20260926 16:44:42.581249 2196 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:42.581416 2196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 16:44:42 dist-test-slave-n1hw krb5kdc[863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441082, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.67@KRBTEST.COM for HTTP/127.21.12.84@KRBTEST.COM
I20260926 16:44:42.612569 2196 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.003s sys 0.000s
I20260926 16:44:42.615281 2214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:42.616722 2196 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:44:42.616854 2196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-2/wal
uuid: "31160480af3f4de1ba0536eda6895a5b"
format_stamp: "Formatted at 2026-09-26 16:44:41 on dist-test-slave-n1hw"
server_key: "ac66df5eccc681f1ade2702ea06c3cb1"
server_key_iv: "5386b03fbc9f2fec0990872d15c8bc8a"
server_key_version: "kuduclusterkey@0"
I20260926 16:44:42.617452 2196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:42.628978 2196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:42.632035 2196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:42.632220 2196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:42.632849 2196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:44:42.633731 2196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:44:42.633817 2196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:42.633883 2196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:44:42.633913 2196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:42.646509 2196 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.67:36321
I20260926 16:44:42.646533 2328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.67:36321 every 8 connection(s)
I20260926 16:44:42.647615 2196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
I20260926 16:44:42.649885 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 2196
Sep 26 16:44:42 dist-test-slave-n1hw krb5kdc[863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441082, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.67@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:44:42.664467 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:42.650424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:35289 (local address 127.21.12.126:34869)
0926 16:44:42.650897 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:42.650902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:42.651405 (+ 503us) server_negotiation.cc:407] Connection header received
0926 16:44:42.654460 (+ 3055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:42.654464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:42.654521 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:42.654595 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:42.655895 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:42.656430 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:42.657873 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:42.658224 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:42.658297 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:42.660630 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:42.660634 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:42.660636 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:42.660661 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:42.662145 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:42.662729 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:42.662733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:42.662735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:42.662834 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:42.663248 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:42.663253 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:42.663255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:42.663476 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:42.663581 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:42.664171 (+ 590us) server_negotiation.cc:299] Negotiation successful
0926 16:44:42.664297 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375}
I20260926 16:44:42.665737 2333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:42.650825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34869 (local address 127.21.12.67:35289)
0926 16:44:42.651263 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:42.651313 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:42.654244 (+ 2931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:42.654776 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:42.654783 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:42.655122 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:42.655757 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:42.655774 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:42.656558 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:42.656562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:42.657704 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:42.657719 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:42.658461 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:42.658464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:42.658597 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:42.659185 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:42.659201 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:42.660477 (+ 1276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:42.662269 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:42.662275 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:42.662289 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:42.662609 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:42.662947 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:42.662951 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:42.662953 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:42.663126 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:42.663595 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:42.663601 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:42.663886 (+ 285us) client_negotiation.cc:769] Sending connection context
0926 16:44:42.664115 (+ 229us) client_negotiation.cc:240] Negotiation successful
0926 16:44:42.664426 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":129,"threads_started":1}
I20260926 16:44:42.667256 2329 heartbeater.cc:347] Connected to a master server at 127.21.12.126:34869
I20260926 16:44:42.667943 2329 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:42.668572 2329 heartbeater.cc:511] Master 127.21.12.126:34869 requested a full tablet report, sending...
W20260926 16:44:42.670334 2329 heartbeater.cc:644] Failed to heartbeat to 127.21.12.126:34869 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 16:44:42.670464 2329 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:42.670650 2329 heartbeater.cc:511] Master 127.21.12.126:34869 requested a full tablet report, sending...
I20260926 16:44:42.671541 2329 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:42.671710 2329 heartbeater.cc:511] Master 127.21.12.126:34869 requested a full tablet report, sending...
W20260926 16:44:42.672555 2329 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 16:44:42.673900 2331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:42.650558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60286 (local address 127.21.12.126:34869)
0926 16:44:42.656942 (+ 6384us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:42.656959 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:42.656973 (+ 14us) server_negotiation.cc:407] Connection header received
0926 16:44:42.657010 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:42.657013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:42.657058 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:42.657162 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:42.659240 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:42.662288 (+ 3048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:42.666986 (+ 4698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:42.667312 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:42.667445 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:42.668983 (+ 1538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:42.668987 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:42.668990 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:42.669014 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:42.672043 (+ 3029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:42.672527 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:42.672530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:42.672532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:42.672628 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:42.672909 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:42.672912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:42.672913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:42.673113 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:42.673321 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:42.673552 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 16:44:42.673670 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6314,"thread_start_us":82,"threads_started":1}
Sep 26 16:44:43 dist-test-slave-n1hw krb5kdc[863](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790441083, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:43.145748 2059 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:43.146095 2059 heartbeater.cc:511] Master 127.21.12.126:34869 requested a full tablet report, sending...
I20260926 16:44:43.147127 1859 ts_manager.cc:194] Registered new tserver with Master: 57c8043747704341923c5c568ce64598 (127.21.12.65:36275)
I20260926 16:44:43.148684 1859 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.65@KRBTEST.COM'} at 127.21.12.65:38793
I20260926 16:44:43.427443 2194 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:43.427743 2194 heartbeater.cc:511] Master 127.21.12.126:34869 requested a full tablet report, sending...
I20260926 16:44:43.428493 1859 ts_manager.cc:194] Registered new tserver with Master: 26cfed3ad4e64a6ebc4ba248d6230505 (127.21.12.66:44209)
I20260926 16:44:43.429323 1859 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.66@KRBTEST.COM'} at 127.21.12.66:40529
Sep 26 16:44:43 dist-test-slave-n1hw krb5kdc[863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441083, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for HTTP/127.21.12.84@KRBTEST.COM
I20260926 16:44:43.672775 2329 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:43.673069 2329 heartbeater.cc:511] Master 127.21.12.126:34869 requested a full tablet report, sending...
I20260926 16:44:43.673902 1859 ts_manager.cc:194] Registered new tserver with Master: 31160480af3f4de1ba0536eda6895a5b (127.21.12.67:36321)
I20260926 16:44:43.674685 1859 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.67@KRBTEST.COM'} at 127.21.12.67:35289
I20260926 16:44:43.676834 21553 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:44:43.689364 2351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:43.679109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60292 (local address 127.21.12.126:34869)
0926 16:44:43.679632 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:43.679637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:43.679952 (+ 315us) server_negotiation.cc:407] Connection header received
0926 16:44:43.679991 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:43.679994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:43.680037 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:43.680104 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:43.680997 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:43.681801 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:43.682984 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:43.683336 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:43.683436 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:43.685252 (+ 1816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:43.685256 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:43.685259 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:43.685286 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:43.687380 (+ 2094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:43.687852 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:43.687857 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:43.687862 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:43.687972 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:43.688267 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:43.688270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:43.688272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:43.688491 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:43.688625 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:43.688897 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 16:44:43.689033 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":87,"threads_started":1}
Sep 26 16:44:43 dist-test-slave-n1hw krb5kdc[863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441083, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for HTTP/127.21.12.84@KRBTEST.COM
I20260926 16:44:43.693336 1859 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60292:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:44:43.698009 1859 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:44:43.721422 2359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:43.711818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:44209 (local address 127.0.0.1:37620)
0926 16:44:43.714571 (+ 2753us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:43.714583 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:43.714658 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:43.716021 (+ 1363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:43.716027 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:43.716050 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:44:43.716177 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:43.716184 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:43.717560 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:43.717564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:43.719589 (+ 2025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:43.719600 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:43.720827 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:43.720831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:43.721018 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:43.721060 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 16:44:43.721162 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 16:44:43.721275 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2696,"thread_start_us":99,"threads_started":1}
I20260926 16:44:43.722010 2360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:43.711987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37620 (local address 127.21.12.66:44209)
0926 16:44:43.713949 (+ 1962us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:43.713956 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:43.714745 (+ 789us) server_negotiation.cc:407] Connection header received
0926 16:44:43.714825 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:43.714830 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:43.715049 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:43.715834 (+ 785us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:44:43.716306 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:43.717433 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:43.719716 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:43.720511 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:43.720623 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:43.720767 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:43.721778 (+ 1011us) server_negotiation.cc:299] Negotiation successful
0926 16:44:43.721876 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1862,"thread_start_us":105,"threads_started":1}
I20260926 16:44:43.726330 2119 tablet_service.cc:1467] Processing CreateTablet for tablet 684ffe23e5eb4ed197296119cc06acc9 (DEFAULT_TABLE table=test-table [id=22955da58afd4aa792c60e91b568f890]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:44:43.727485 2119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 684ffe23e5eb4ed197296119cc06acc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:44:43.728425 2359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:43.713029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:36275 (local address 127.0.0.1:55902)
0926 16:44:43.722354 (+ 9325us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:43.722366 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:43.722498 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:43.724095 (+ 1597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:43.724097 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:43.724115 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:44:43.724179 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:43.724184 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:43.725603 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:43.725607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:43.726776 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:43.726788 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:43.728048 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:43.728052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:43.728147 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:43.728169 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 16:44:43.728241 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 16:44:43.728301 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9275,"thread_start_us":87,"threads_started":1}
I20260926 16:44:43.728513 2363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:43.713394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55902 (local address 127.21.12.65:36275)
0926 16:44:43.713687 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:43.713694 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:43.722432 (+ 8738us) server_negotiation.cc:407] Connection header received
0926 16:44:43.722838 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:43.722843 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:43.723047 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:43.723158 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:44:43.724332 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:43.725447 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:43.726931 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:43.727735 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:43.727814 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:43.727955 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:43.728283 (+ 328us) server_negotiation.cc:299] Negotiation successful
0926 16:44:43.728378 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":110,"threads_started":1}
I20260926 16:44:43.731207 1984 tablet_service.cc:1467] Processing CreateTablet for tablet 684ffe23e5eb4ed197296119cc06acc9 (DEFAULT_TABLE table=test-table [id=22955da58afd4aa792c60e91b568f890]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:44:43.732301 1984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 684ffe23e5eb4ed197296119cc06acc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:44:43.734480 2364 tablet_bootstrap.cc:492] T 684ffe23e5eb4ed197296119cc06acc9 P 26cfed3ad4e64a6ebc4ba248d6230505: Bootstrap starting.
I20260926 16:44:43.739897 2361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:43.710794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.67:36321 (local address 127.0.0.1:41476)
0926 16:44:43.714191 (+ 3397us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:43.714239 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:43.714359 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:43.722230 (+ 7871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:43.722233 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:43.722251 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:44:43.722333 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:43.722338 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:43.735942 (+ 13604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:43.735945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:43.737211 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:43.737223 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:43.739474 (+ 2251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:43.739478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:43.739573 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:43.739612 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 16:44:43.739667 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 16:44:43.739747 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3226,"thread_start_us":111,"threads_started":1}
I20260926 16:44:43.740553 2358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:43.710925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41476 (local address 127.21.12.67:36321)
0926 16:44:43.721587 (+ 10662us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:43.721596 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:43.721622 (+ 26us) server_negotiation.cc:407] Connection header received
0926 16:44:43.721694 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:43.721700 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:43.721923 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:43.722066 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:44:43.722454 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:43.735755 (+ 13301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:43.737366 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:43.739323 (+ 1957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:43.740102 (+ 779us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:43.740246 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:43.740300 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 16:44:43.740391 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10561,"thread_start_us":79,"threads_started":1}
I20260926 16:44:43.741302 2365 tablet_bootstrap.cc:492] T 684ffe23e5eb4ed197296119cc06acc9 P 57c8043747704341923c5c568ce64598: Bootstrap starting.
I20260926 16:44:43.742453 2364 tablet_bootstrap.cc:654] T 684ffe23e5eb4ed197296119cc06acc9 P 26cfed3ad4e64a6ebc4ba248d6230505: Neither blocks nor log segments found. Creating new log.
I20260926 16:44:43.743290 2364 log.cc:824] T 684ffe23e5eb4ed197296119cc06acc9 P 26cfed3ad4e64a6ebc4ba248d6230505: Log is configured to *not* fsync() on all Append() calls
I20260926 16:44:43.743395 2365 tablet_bootstrap.cc:654] T 684ffe23e5eb4ed197296119cc06acc9 P 57c8043747704341923c5c568ce64598: Neither blocks nor log segments found. Creating new log.
I20260926 16:44:43.743606 2263 tablet_service.cc:1467] Processing CreateTablet for tablet 684ffe23e5eb4ed197296119cc06acc9 (DEFAULT_TABLE table=test-table [id=22955da58afd4aa792c60e91b568f890]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:44:43.744204 2365 log.cc:824] T 684ffe23e5eb4ed197296119cc06acc9 P 57c8043747704341923c5c568ce64598: Log is configured to *not* fsync() on all Append() calls
I20260926 16:44:43.744630 2263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 684ffe23e5eb4ed197296119cc06acc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:44:43.746992 2365 tablet_bootstrap.cc:492] T 684ffe23e5eb4ed197296119cc06acc9 P 57c8043747704341923c5c568ce64598: No bootstrap required, opened a new log
I20260926 16:44:43.747186 2365 ts_tablet_manager.cc:1397] T 684ffe23e5eb4ed197296119cc06acc9 P 57c8043747704341923c5c568ce64598: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 16:44:43.747430 2364 tablet_bootstrap.cc:492] T 684ffe23e5eb4ed197296119cc06acc9 P 26cfed3ad4e64a6ebc4ba248d6230505: No bootstrap required, opened a new log
I20260926 16:44:43.747655 2364 ts_tablet_manager.cc:1397] T 684ffe23e5eb4ed197296119cc06acc9 P 26cfed3ad4e64a6ebc4ba248d6230505: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20260926 16:44:43.751044 2365 raft_consensus.cc:348] T 684ffe23e5eb4ed197296119cc06acc9 P 57c8043747704341923c5c568ce64598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31160480af3f4de1ba0536eda6895a5b" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36321 } } peers { permanent_uuid: "26cfed3ad4e64a6ebc4ba248d6230505" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 44209 } } peers { permanent_uuid: "57c8043747704341923c5c568ce64598" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 36275 } }
I20260926 16:44:43.751292 2365 raft_consensus.cc:374] T 684ffe23e5eb4ed197296119cc06acc9 P 57c8043747704341923c5c568ce64598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:44:43.751348 2365 raft_consensus.cc:729] T 684ffe23e5eb4ed197296119cc06acc9 P 57c8043747704341923c5c568ce64598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57c8043747704341923c5c568ce64598, State: Initialized, Role: FOLLOWER
I20260926 16:44:43.751528 2364 raft_consensus.cc:348] T 684ffe23e5eb4ed197296119cc06acc9 P 26cfed3ad4e64a6ebc4ba248d6230505 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31160480af3f4de1ba0536eda6895a5b" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36321 } } peers { permanent_uuid: "26cfed3ad4e64a6ebc4ba248d6230505" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 44209 } } peers { permanent_uuid: "57c8043747704341923c5c568ce64598" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 36275 } }
I20260926 16:44:43.751756 2364 raft_consensus.cc:374] T 684ffe23e5eb4ed197296119cc06acc9 P 26cfed3ad4e64a6ebc4ba248d6230505 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:44:43.751801 2364 raft_consensus.cc:729] T 684ffe23e5eb4ed197296119cc06acc9 P 26cfed3ad4e64a6ebc4ba248d6230505 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26cfed3ad4e64a6ebc4ba248d6230505, State: Initialized, Role: FOLLOWER
I20260926 16:44:43.751797 2365 consensus_queue.cc:260] T 684ffe23e5eb4ed197296119cc06acc9 P 57c8043747704341923c5c568ce64598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31160480af3f4de1ba0536eda6895a5b" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36321 } } peers { permanent_uuid: "26cfed3ad4e64a6ebc4ba248d6230505" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 44209 } } peers { permanent_uuid: "57c8043747704341923c5c568ce64598" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 36275 } }
I20260926 16:44:43.752298 2364 consensus_queue.cc:260] T 684ffe23e5eb4ed197296119cc06acc9 P 26cfed3ad4e64a6ebc4ba248d6230505 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31160480af3f4de1ba0536eda6895a5b" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36321 } } peers { permanent_uuid: "26cfed3ad4e64a6ebc4ba248d6230505" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 44209 } } peers { permanent_uuid: "57c8043747704341923c5c568ce64598" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 36275 } }
I20260926 16:44:43.752792 2365 ts_tablet_manager.cc:1428] T 684ffe23e5eb4ed197296119cc06acc9 P 57c8043747704341923c5c568ce64598: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 16:44:43.753139 2059 heartbeater.cc:503] Master 127.21.12.126:34869 was elected leader, sending a full tablet report...
I20260926 16:44:43.753714 2364 ts_tablet_manager.cc:1428] T 684ffe23e5eb4ed197296119cc06acc9 P 26cfed3ad4e64a6ebc4ba248d6230505: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 16:44:43.754441 2370 tablet_bootstrap.cc:492] T 684ffe23e5eb4ed197296119cc06acc9 P 31160480af3f4de1ba0536eda6895a5b: Bootstrap starting.
I20260926 16:44:43.755313 2194 heartbeater.cc:503] Master 127.21.12.126:34869 was elected leader, sending a full tablet report...
I20260926 16:44:43.757927 2370 tablet_bootstrap.cc:654] T 684ffe23e5eb4ed197296119cc06acc9 P 31160480af3f4de1ba0536eda6895a5b: Neither blocks nor log segments found. Creating new log.
I20260926 16:44:43.759351 2370 log.cc:824] T 684ffe23e5eb4ed197296119cc06acc9 P 31160480af3f4de1ba0536eda6895a5b: Log is configured to *not* fsync() on all Append() calls
I20260926 16:44:43.763545 2370 tablet_bootstrap.cc:492] T 684ffe23e5eb4ed197296119cc06acc9 P 31160480af3f4de1ba0536eda6895a5b: No bootstrap required, opened a new log
I20260926 16:44:43.763784 2370 ts_tablet_manager.cc:1397] T 684ffe23e5eb4ed197296119cc06acc9 P 31160480af3f4de1ba0536eda6895a5b: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.003s
I20260926 16:44:43.768720 2370 raft_consensus.cc:348] T 684ffe23e5eb4ed197296119cc06acc9 P 31160480af3f4de1ba0536eda6895a5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31160480af3f4de1ba0536eda6895a5b" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36321 } } peers { permanent_uuid: "26cfed3ad4e64a6ebc4ba248d6230505" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 44209 } } peers { permanent_uuid: "57c8043747704341923c5c568ce64598" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 36275 } }
I20260926 16:44:43.769073 2370 raft_consensus.cc:374] T 684ffe23e5eb4ed197296119cc06acc9 P 31160480af3f4de1ba0536eda6895a5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:44:43.769167 2370 raft_consensus.cc:729] T 684ffe23e5eb4ed197296119cc06acc9 P 31160480af3f4de1ba0536eda6895a5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31160480af3f4de1ba0536eda6895a5b, State: Initialized, Role: FOLLOWER
I20260926 16:44:43.769781 2370 consensus_queue.cc:260] T 684ffe23e5eb4ed197296119cc06acc9 P 31160480af3f4de1ba0536eda6895a5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31160480af3f4de1ba0536eda6895a5b" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36321 } } peers { permanent_uuid: "26cfed3ad4e64a6ebc4ba248d6230505" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 44209 } } peers { permanent_uuid: "57c8043747704341923c5c568ce64598" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 36275 } }
I20260926 16:44:43.770936 2370 ts_tablet_manager.cc:1428] T 684ffe23e5eb4ed197296119cc06acc9 P 31160480af3f4de1ba0536eda6895a5b: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260926 16:44:43.771325 2329 heartbeater.cc:503] Master 127.21.12.126:34869 was elected leader, sending a full tablet report...
W20260926 16:44:43.870033 2060 tablet.cc:2369] T 684ffe23e5eb4ed197296119cc06acc9 P 57c8043747704341923c5c568ce64598: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:44:43.888809 2195 tablet.cc:2369] T 684ffe23e5eb4ed197296119cc06acc9 P 26cfed3ad4e64a6ebc4ba248d6230505: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:44:43.899724 2330 tablet.cc:2369] T 684ffe23e5eb4ed197296119cc06acc9 P 31160480af3f4de1ba0536eda6895a5b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:44:43.968658 2369 raft_consensus.cc:482] T 684ffe23e5eb4ed197296119cc06acc9 P 26cfed3ad4e64a6ebc4ba248d6230505 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:44:43.968886 2369 raft_consensus.cc:504] T 684ffe23e5eb4ed197296119cc06acc9 P 26cfed3ad4e64a6ebc4ba248d6230505 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31160480af3f4de1ba0536eda6895a5b" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36321 } } peers { permanent_uuid: "26cfed3ad4e64a6ebc4ba248d6230505" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 44209 } } peers { permanent_uuid: "57c8043747704341923c5c568ce64598" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 36275 } }
I20260926 16:44:43.970427 2369 leader_election.cc:288] T 684ffe23e5eb4ed197296119cc06acc9 P 26cfed3ad4e64a6ebc4ba248d6230505 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31160480af3f4de1ba0536eda6895a5b (127.21.12.67:36321), 57c8043747704341923c5c568ce64598 (127.21.12.65:36275)
I20260926 16:44:43.976279 2358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:43.971113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:55345 (local address 127.21.12.67:36321)
0926 16:44:43.971236 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:43.971240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:43.971575 (+ 335us) server_negotiation.cc:407] Connection header received
0926 16:44:43.971747 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:43.971750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:43.971799 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:43.971846 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:44:43.972312 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:43.973126 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:43.974893 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:43.975500 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:43.975598 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:43.975659 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:43.976028 (+ 369us) server_negotiation.cc:299] Negotiation successful
0926 16:44:43.976118 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 16:44:43.976768 2363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:43.971148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:49385 (local address 127.21.12.65:36275)
0926 16:44:43.971264 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:43.971269 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:43.972709 (+ 1440us) server_negotiation.cc:407] Connection header received
0926 16:44:43.972886 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:43.972890 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:43.972941 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:43.973006 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:44:43.973389 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:43.974145 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:43.975530 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:43.976130 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:43.976218 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:43.976264 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:43.976579 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 16:44:43.976654 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 16:44:43.977177 2375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:43.970750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:36275 (local address 127.21.12.66:49385)
0926 16:44:43.972624 (+ 1874us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:43.972638 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:43.972774 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:43.973139 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:43.973141 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:43.973157 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:44:43.973269 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:43.973275 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:43.974286 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:43.974289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:43.975389 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:43.975400 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:43.976354 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:43.976357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:43.976452 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:43.976473 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 16:44:43.976967 (+ 494us) client_negotiation.cc:240] Negotiation successful
0926 16:44:43.977040 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1809,"thread_start_us":101,"threads_started":1}
I20260926 16:44:43.977164 2374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:43.970687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.67:36321 (local address 127.21.12.66:55345)
0926 16:44:43.971467 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:43.971482 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:43.971603 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:43.972003 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:43.972006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:43.972025 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:44:43.972167 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:43.972173 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:43.973277 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:43.973281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:43.974744 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:43.974756 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:43.975755 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:43.975758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:43.975864 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:43.975886 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 16:44:43.976732 (+ 846us) client_negotiation.cc:240] Negotiation successful
0926 16:44:43.976982 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"thread_start_us":117,"threads_started":1}
I20260926 16:44:43.978009 2008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "684ffe23e5eb4ed197296119cc06acc9" candidate_uuid: "26cfed3ad4e64a6ebc4ba248d6230505" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57c8043747704341923c5c568ce64598" is_pre_election: true
I20260926 16:44:43.978243 2270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "684ffe23e5eb4ed197296119cc06acc9" candidate_uuid: "26cfed3ad4e64a6ebc4ba248d6230505" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31160480af3f4de1ba0536eda6895a5b" is_pre_election: true
I20260926 16:44:43.978331 2008 raft_consensus.cc:2454] T 684ffe23e5eb4ed197296119cc06acc9 P 57c8043747704341923c5c568ce64598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26cfed3ad4e64a6ebc4ba248d6230505 in term 0.
I20260926 16:44:43.978528 2270 raft_consensus.cc:2454] T 684ffe23e5eb4ed197296119cc06acc9 P 31160480af3f4de1ba0536eda6895a5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26cfed3ad4e64a6ebc4ba248d6230505 in term 0.
I20260926 16:44:43.978785 2080 leader_election.cc:302] T 684ffe23e5eb4ed197296119cc06acc9 P 26cfed3ad4e64a6ebc4ba248d6230505 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26cfed3ad4e64a6ebc4ba248d6230505, 57c8043747704341923c5c568ce64598; no voters:
I20260926 16:44:43.979089 2369 raft_consensus.cc:2781] T 684ffe23e5eb4ed197296119cc06acc9 P 26cfed3ad4e64a6ebc4ba248d6230505 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:44:43.979159 2369 raft_consensus.cc:482] T 684ffe23e5eb4ed197296119cc06acc9 P 26cfed3ad4e64a6ebc4ba248d6230505 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:44:43.979198 2369 raft_consensus.cc:3037] T 684ffe23e5eb4ed197296119cc06acc9 P 26cfed3ad4e64a6ebc4ba248d6230505 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:44:43.980494 2369 raft_consensus.cc:504] T 684ffe23e5eb4ed197296119cc06acc9 P 26cfed3ad4e64a6ebc4ba248d6230505 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31160480af3f4de1ba0536eda6895a5b" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36321 } } peers { permanent_uuid: "26cfed3ad4e64a6ebc4ba248d6230505" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 44209 } } peers { permanent_uuid: "57c8043747704341923c5c568ce64598" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 36275 } }
I20260926 16:44:43.980883 2369 leader_election.cc:288] T 684ffe23e5eb4ed197296119cc06acc9 P 26cfed3ad4e64a6ebc4ba248d6230505 [CANDIDATE]: Term 1 election: Requested vote from peers 31160480af3f4de1ba0536eda6895a5b (127.21.12.67:36321), 57c8043747704341923c5c568ce64598 (127.21.12.65:36275)
I20260926 16:44:43.981362 2008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "684ffe23e5eb4ed197296119cc06acc9" candidate_uuid: "26cfed3ad4e64a6ebc4ba248d6230505" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57c8043747704341923c5c568ce64598"
I20260926 16:44:43.981478 2008 raft_consensus.cc:3037] T 684ffe23e5eb4ed197296119cc06acc9 P 57c8043747704341923c5c568ce64598 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:44:43.982774 2008 raft_consensus.cc:2454] T 684ffe23e5eb4ed197296119cc06acc9 P 57c8043747704341923c5c568ce64598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26cfed3ad4e64a6ebc4ba248d6230505 in term 1.
I20260926 16:44:43.983114 2080 leader_election.cc:302] T 684ffe23e5eb4ed197296119cc06acc9 P 26cfed3ad4e64a6ebc4ba248d6230505 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26cfed3ad4e64a6ebc4ba248d6230505, 57c8043747704341923c5c568ce64598; no voters:
I20260926 16:44:43.983629 2270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "684ffe23e5eb4ed197296119cc06acc9" candidate_uuid: "26cfed3ad4e64a6ebc4ba248d6230505" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31160480af3f4de1ba0536eda6895a5b"
I20260926 16:44:43.983752 2270 raft_consensus.cc:3037] T 684ffe23e5eb4ed197296119cc06acc9 P 31160480af3f4de1ba0536eda6895a5b [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:44:43.984916 2270 raft_consensus.cc:2454] T 684ffe23e5eb4ed197296119cc06acc9 P 31160480af3f4de1ba0536eda6895a5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26cfed3ad4e64a6ebc4ba248d6230505 in term 1.
I20260926 16:44:43.985440 2369 raft_consensus.cc:2781] T 684ffe23e5eb4ed197296119cc06acc9 P 26cfed3ad4e64a6ebc4ba248d6230505 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:44:43.985657 2369 raft_consensus.cc:686] T 684ffe23e5eb4ed197296119cc06acc9 P 26cfed3ad4e64a6ebc4ba248d6230505 [term 1 LEADER]: Becoming Leader. State: Replica: 26cfed3ad4e64a6ebc4ba248d6230505, State: Running, Role: LEADER
I20260926 16:44:43.986155 2369 consensus_queue.cc:237] T 684ffe23e5eb4ed197296119cc06acc9 P 26cfed3ad4e64a6ebc4ba248d6230505 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31160480af3f4de1ba0536eda6895a5b" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36321 } } peers { permanent_uuid: "26cfed3ad4e64a6ebc4ba248d6230505" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 44209 } } peers { permanent_uuid: "57c8043747704341923c5c568ce64598" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 36275 } }
I20260926 16:44:43.989632 1858 catalog_manager.cc:5495] T 684ffe23e5eb4ed197296119cc06acc9 P 26cfed3ad4e64a6ebc4ba248d6230505 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26cfed3ad4e64a6ebc4ba248d6230505 (127.21.12.66). New cstate: current_term: 1 leader_uuid: "26cfed3ad4e64a6ebc4ba248d6230505" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31160480af3f4de1ba0536eda6895a5b" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26cfed3ad4e64a6ebc4ba248d6230505" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 44209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57c8043747704341923c5c568ce64598" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 36275 } health_report { overall_health: UNKNOWN } } }
I20260926 16:44:44.008240 2360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:44.003054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37636 (local address 127.21.12.66:44209)
0926 16:44:44.003174 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:44.003178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:44.003193 (+ 15us) server_negotiation.cc:407] Connection header received
0926 16:44:44.003272 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:44.003275 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:44.003338 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:44.003458 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:44:44.003849 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:44.004630 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:44.005918 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:44.006272 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:44.006382 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:44.006828 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:44:44.007381 (+ 553us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:44:44.007513 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:44.008004 (+ 491us) server_negotiation.cc:299] Negotiation successful
0926 16:44:44.008069 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 16:44:44.015977 2008 raft_consensus.cc:1261] T 684ffe23e5eb4ed197296119cc06acc9 P 57c8043747704341923c5c568ce64598 [term 1 FOLLOWER]: Refusing update from remote peer 26cfed3ad4e64a6ebc4ba248d6230505: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:44:44.016824 2376 consensus_queue.cc:1035] T 684ffe23e5eb4ed197296119cc06acc9 P 26cfed3ad4e64a6ebc4ba248d6230505 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57c8043747704341923c5c568ce64598" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 36275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:44:44.023700 2270 raft_consensus.cc:1261] T 684ffe23e5eb4ed197296119cc06acc9 P 31160480af3f4de1ba0536eda6895a5b [term 1 FOLLOWER]: Refusing update from remote peer 26cfed3ad4e64a6ebc4ba248d6230505: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:44:44.024367 2376 consensus_queue.cc:1035] T 684ffe23e5eb4ed197296119cc06acc9 P 26cfed3ad4e64a6ebc4ba248d6230505 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31160480af3f4de1ba0536eda6895a5b" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:44:44.031805 2390 mvcc.cc:205] Tried to move back new op lower bound from 7333646680120713216 to 7333646680011276288. Current Snapshot: MvccSnapshot[applied={T|T < 7333646680120713216}]
I20260926 16:44:44.033741 2387 mvcc.cc:205] Tried to move back new op lower bound from 7333646680120713216 to 7333646680011276288. Current Snapshot: MvccSnapshot[applied={T|T < 7333646680120713216}]
I20260926 16:44:44.035576 2381 mvcc.cc:205] Tried to move back new op lower bound from 7333646680120713216 to 7333646680011276288. Current Snapshot: MvccSnapshot[applied={T|T < 7333646680120713216}]
I20260926 16:44:44.044792 1859 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:44:44.044986 1859 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:44:44.047652 1859 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 52665a9e0d8b43a09dcc7d0f18d87b8e: Sending DeleteTablet for 3 replicas of tablet 684ffe23e5eb4ed197296119cc06acc9
I20260926 16:44:44.048563 2263 tablet_service.cc:1514] Processing DeleteTablet for tablet 684ffe23e5eb4ed197296119cc06acc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:44:44 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:41476
I20260926 16:44:44.048624 1984 tablet_service.cc:1514] Processing DeleteTablet for tablet 684ffe23e5eb4ed197296119cc06acc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:44:44 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:55902
I20260926 16:44:44.048727 2119 tablet_service.cc:1514] Processing DeleteTablet for tablet 684ffe23e5eb4ed197296119cc06acc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:44:44 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:37620
I20260926 16:44:44.049793 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 1897
I20260926 16:44:44.053989 2396 tablet_replica.cc:330] T 684ffe23e5eb4ed197296119cc06acc9 P 31160480af3f4de1ba0536eda6895a5b: stopping tablet replica
I20260926 16:44:44.054324 2396 raft_consensus.cc:2229] T 684ffe23e5eb4ed197296119cc06acc9 P 31160480af3f4de1ba0536eda6895a5b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:44:44.054661 2396 raft_consensus.cc:2258] T 684ffe23e5eb4ed197296119cc06acc9 P 31160480af3f4de1ba0536eda6895a5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:44:44.055017 2397 tablet_replica.cc:330] T 684ffe23e5eb4ed197296119cc06acc9 P 26cfed3ad4e64a6ebc4ba248d6230505: stopping tablet replica
I20260926 16:44:44.055284 2397 raft_consensus.cc:2229] T 684ffe23e5eb4ed197296119cc06acc9 P 26cfed3ad4e64a6ebc4ba248d6230505 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:44:44.055647 2397 raft_consensus.cc:2258] T 684ffe23e5eb4ed197296119cc06acc9 P 26cfed3ad4e64a6ebc4ba248d6230505 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:44:44.057013 2396 ts_tablet_manager.cc:1905] T 684ffe23e5eb4ed197296119cc06acc9 P 31160480af3f4de1ba0536eda6895a5b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:44:44.059257 2396 ts_tablet_manager.cc:1918] T 684ffe23e5eb4ed197296119cc06acc9 P 31160480af3f4de1ba0536eda6895a5b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:44:44.059369 2396 log.cc:1196] T 684ffe23e5eb4ed197296119cc06acc9 P 31160480af3f4de1ba0536eda6895a5b: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-2/wal/wals/684ffe23e5eb4ed197296119cc06acc9
I20260926 16:44:44.059686 2396 ts_tablet_manager.cc:1939] T 684ffe23e5eb4ed197296119cc06acc9 P 31160480af3f4de1ba0536eda6895a5b: Deleting consensus metadata
I20260926 16:44:44.060922 1846 catalog_manager.cc:4841] TS 31160480af3f4de1ba0536eda6895a5b (127.21.12.67:36321): tablet 684ffe23e5eb4ed197296119cc06acc9 (table test-table [id=22955da58afd4aa792c60e91b568f890]) successfully deleted
I20260926 16:44:44.062144 2397 ts_tablet_manager.cc:1905] T 684ffe23e5eb4ed197296119cc06acc9 P 26cfed3ad4e64a6ebc4ba248d6230505: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:44:44.064822 2397 ts_tablet_manager.cc:1918] T 684ffe23e5eb4ed197296119cc06acc9 P 26cfed3ad4e64a6ebc4ba248d6230505: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:44:44.064921 2397 log.cc:1196] T 684ffe23e5eb4ed197296119cc06acc9 P 26cfed3ad4e64a6ebc4ba248d6230505: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440924937351-21553-0/minicluster-data/ts-1/wal/wals/684ffe23e5eb4ed197296119cc06acc9
I20260926 16:44:44.065325 2397 ts_tablet_manager.cc:1939] T 684ffe23e5eb4ed197296119cc06acc9 P 26cfed3ad4e64a6ebc4ba248d6230505: Deleting consensus metadata
W20260926 16:44:44.065379 1843 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.12.65:36275 (error 108)
I20260926 16:44:44.066594 1846 catalog_manager.cc:4841] TS 26cfed3ad4e64a6ebc4ba248d6230505 (127.21.12.66:44209): tablet 684ffe23e5eb4ed197296119cc06acc9 (table test-table [id=22955da58afd4aa792c60e91b568f890]) successfully deleted
I20260926 16:44:44.066922 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 2063
I20260926 16:44:44.067001 2361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:44.066688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:36275 (local address 127.0.0.1:55916)
0926 16:44:44.066808 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:44.066886 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.65:36275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 16:44:44.068486 1843 catalog_manager.cc:4584] TS 57c8043747704341923c5c568ce64598 (127.21.12.65:36275): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 684ffe23e5eb4ed197296119cc06acc9: Network error: Client connection negotiation failed: client connection to 127.21.12.65:36275: connect: Connection refused (error 111)
I20260926 16:44:44.074259 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 2196
I20260926 16:44:44.080021 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 1826
I20260926 16:44:44.088035 21553 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 16:44:44.422156 21553 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 16:44:44.422284 21553 mini_ranger.cc:66] Stopping Ranger...
I20260926 16:44:44.766413 21553 mini_ranger.cc:68] Stopped Ranger
2026-09-26 16:44:44.766 UTC [893] LOG: received smart shutdown request
2026-09-26 16:44:44.770 UTC [893] LOG: background worker "logical replication launcher" (PID 902) exited with exit code 1
2026-09-26 16:44:44.771 UTC [897] LOG: shutting down
2026-09-26 16:44:44.949 UTC [893] LOG: database system is shut down
2026-09-26T16:44:44Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48651 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:44:45 dist-test-slave-n1hw krb5kdc[2407](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:44:45 dist-test-slave-n1hw krb5kdc[2407](info): set up 2 sockets
Sep 26 16:44:45 dist-test-slave-n1hw krb5kdc[2407](info): commencing operation
krb5kdc: starting...
W20260926 16:44:46.100059 21553 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Sep 26 16:44:46 dist-test-slave-n1hw krb5kdc[2407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:44:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:44:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
I20260926 16:44:46.216279 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:46021
--webserver_interface=127.21.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:44591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:46021,127.21.12.125:39091,127.21.12.124:38779
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:44:46.325825 2420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:46.326095 2420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:46.326192 2420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:46.330135 2420 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:44:46.330211 2420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:46.330247 2420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:46.330294 2420 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:44:46.330331 2420 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:44:46.335542 2420 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:44591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:46021,127.21.12.125:39091,127.21.12.124:38779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:46021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:44:46.336556 2420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:46.337592 2420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:46.345705 2425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:46.345777 2428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:46.345893 2420 server_base.cc:1034] running on GCE node
W20260926 16:44:46.345662 2426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:46.346604 2420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:46.347587 2420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:46.348774 2420 hybrid_clock.cc:648] HybridClock initialized: now 1790441086348760 us; error 24 us; skew 500 ppm
Sep 26 16:44:46 dist-test-slave-n1hw krb5kdc[2407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441086, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:46.351426 2420 init.cc:377] Logged in from keytab as kudu/127.21.12.126@KRBTEST.COM (short username kudu)
I20260926 16:44:46.352288 2420 webserver.cc:460] Webserver started at http://127.21.12.126:35355/ using document root <none> and password file <none>
I20260926 16:44:46.352807 2420 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:46.352891 2420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:46.353112 2420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:44:46.355131 2420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "1db1f8e5fc714522af7aabbc0545a17c"
format_stamp: "Formatted at 2026-09-26 16:44:46 on dist-test-slave-n1hw"
server_key: "2bdad0695c71e66eb2a7a6a9184e2cd9"
server_key_iv: "1c0e060b7042b42368b10bad71271880"
server_key_version: "encryptionkey@0"
I20260926 16:44:46.355670 2420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "1db1f8e5fc714522af7aabbc0545a17c"
format_stamp: "Formatted at 2026-09-26 16:44:46 on dist-test-slave-n1hw"
server_key: "2bdad0695c71e66eb2a7a6a9184e2cd9"
server_key_iv: "1c0e060b7042b42368b10bad71271880"
server_key_version: "encryptionkey@0"
I20260926 16:44:46.359197 2420 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:44:46.361419 2435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:46.362622 2420 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:44:46.362748 2420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "1db1f8e5fc714522af7aabbc0545a17c"
format_stamp: "Formatted at 2026-09-26 16:44:46 on dist-test-slave-n1hw"
server_key: "2bdad0695c71e66eb2a7a6a9184e2cd9"
server_key_iv: "1c0e060b7042b42368b10bad71271880"
server_key_version: "encryptionkey@0"
I20260926 16:44:46.362847 2420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:46.400369 2420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:46.403232 2420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:46.403429 2420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:46.410228 2420 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.126:46021
I20260926 16:44:46.410247 2487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.126:46021 every 8 connection(s)
I20260926 16:44:46.411109 2420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:44:46.411808 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 2420
I20260926 16:44:46.411893 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:44:46.412145 21553 external_mini_cluster.cc:1428] Setting key 01f0fa43765bcc44988d8c83326406f3
I20260926 16:44:46.413818 2488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:44:46.417361 2488 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46021 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.12.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.125@KRBTEST.COM" created.
Sep 26 16:44:46 dist-test-slave-n1hw krb5kdc[2407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441086, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:44:46.433166 2492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:46.419552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40676 (local address 127.21.12.126:46021)
0926 16:44:46.419967 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:46.419987 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:46.420329 (+ 342us) server_negotiation.cc:407] Connection header received
0926 16:44:46.421355 (+ 1026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:46.421360 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:46.421545 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:46.421851 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:46.422936 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.424315 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.425327 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.426033 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.426205 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:46.428516 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:46.428526 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:46.428545 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:46.428571 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:46.430389 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:46.430912 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:46.430917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:46.430918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:46.431022 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:46.431354 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:46.431357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:46.431358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:46.431768 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:46.431911 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:46.432228 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 16:44:46.432380 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":286,"spinlock_wait_cycles":37632,"thread_start_us":156,"threads_started":1}
I20260926 16:44:46.433172 2491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:46.419552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:46021 (local address 127.0.0.1:40676)
0926 16:44:46.420153 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:46.420202 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:46.421072 (+ 870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:46.421873 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:46.421884 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:46.422305 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:46.422749 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:46.422770 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.424465 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.424468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:46.425204 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:46.425213 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.426184 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.426187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:46.426311 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:46.426932 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:46.426948 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:46.428359 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:46.430554 (+ 2195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:46.430566 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:46.430579 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:46.430812 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:46.431113 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:46.431116 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:46.431120 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:46.431270 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:46.431882 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:46.431887 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:46.432007 (+ 120us) client_negotiation.cc:769] Sending connection context
0926 16:44:46.432145 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 16:44:46.432469 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":136,"thread_start_us":139,"threads_started":1}
I20260926 16:44:46.435539 2488 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.125" port: 39091 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.125.keytab.
Entry for principal kudu/127.21.12.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.125.keytab.
I20260926 16:44:46.436306 2491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:46.435981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.125:39091 (local address 127.0.0.1:52862)
0926 16:44:46.436139 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:46.436217 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.125:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260926 16:44:46.436527 2438 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.12.125:39091: connect: Connection refused (error 111)
I20260926 16:44:46.437021 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.125:39091
--webserver_interface=127.21.12.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.125.keytab
--principal=kudu/127.21.12.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:44591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:46021,127.21.12.125:39091,127.21.12.124:38779
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:44:46.437368 2491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:46.437151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.125:39091 (local address 127.0.0.1:52874)
0926 16:44:46.437284 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:46.437322 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.125:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260926 16:44:46.437584 2488 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.125:39091: Network error: Client connection negotiation failed: client connection to 127.21.12.125:39091: connect: Connection refused (error 111)
I20260926 16:44:46.486860 2488 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.125" port: 39091 } attempt: 1
I20260926 16:44:46.487709 2491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:46.487401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.125:39091 (local address 127.0.0.1:52888)
0926 16:44:46.487557 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:46.487635 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.125:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 16:44:46.488464 2491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:46.488258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.125:39091 (local address 127.0.0.1:52904)
0926 16:44:46.488356 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:46.488401 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.125:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 16:44:46.488621 2488 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.125:39091: Network error: Client connection negotiation failed: client connection to 127.21.12.125:39091: connect: Connection refused (error 111)
W20260926 16:44:46.544697 2496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:46.544915 2496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:46.544953 2496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:46.548965 2496 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:44:46.549022 2496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:46.549042 2496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:46.549060 2496 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:44:46.549075 2496 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:44:46.554191 2496 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:44591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:46021,127.21.12.125:39091,127.21.12.124:38779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.125.keytab
--principal=kudu/127.21.12.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.125:39091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.12.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:44:46.555130 2496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:46.556079 2496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 16:44:46.556789 2488 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.125" port: 39091 } attempt: 2
I20260926 16:44:46.557601 2491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:46.557324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.125:39091 (local address 127.0.0.1:52912)
0926 16:44:46.557470 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:46.557528 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.125:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 16:44:46.558284 2491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:46.558106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.125:39091 (local address 127.0.0.1:52928)
0926 16:44:46.558194 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:46.558235 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.125:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 16:44:46.558418 2488 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.125:39091: Network error: Client connection negotiation failed: client connection to 127.21.12.125:39091: connect: Connection refused (error 111)
W20260926 16:44:46.564409 2502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:46.564400 2501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:46.564448 2504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:46.564643 2496 server_base.cc:1034] running on GCE node
I20260926 16:44:46.565258 2496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:46.566430 2496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:46.567663 2496 hybrid_clock.cc:648] HybridClock initialized: now 1790441086567655 us; error 29 us; skew 500 ppm
Sep 26 16:44:46 dist-test-slave-n1hw krb5kdc[2407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441086, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:46.570377 2496 init.cc:377] Logged in from keytab as kudu/127.21.12.125@KRBTEST.COM (short username kudu)
I20260926 16:44:46.571204 2496 webserver.cc:460] Webserver started at http://127.21.12.125:40001/ using document root <none> and password file <none>
I20260926 16:44:46.571723 2496 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:46.571780 2496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:46.571981 2496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:44:46.573998 2496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-1/data/instance:
uuid: "957dca54515a4597892043c904a5fcad"
format_stamp: "Formatted at 2026-09-26 16:44:46 on dist-test-slave-n1hw"
server_key: "57211a14a678e34a9a031ceb0547ed23"
server_key_iv: "0e524ed2e12239e6e750874a25f27bc7"
server_key_version: "encryptionkey@0"
I20260926 16:44:46.574539 2496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-1/wal/instance:
uuid: "957dca54515a4597892043c904a5fcad"
format_stamp: "Formatted at 2026-09-26 16:44:46 on dist-test-slave-n1hw"
server_key: "57211a14a678e34a9a031ceb0547ed23"
server_key_iv: "0e524ed2e12239e6e750874a25f27bc7"
server_key_version: "encryptionkey@0"
I20260926 16:44:46.578145 2496 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:44:46.580562 2511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:46.581689 2496 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:44:46.581854 2496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-1/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-1/wal
uuid: "957dca54515a4597892043c904a5fcad"
format_stamp: "Formatted at 2026-09-26 16:44:46 on dist-test-slave-n1hw"
server_key: "57211a14a678e34a9a031ceb0547ed23"
server_key_iv: "0e524ed2e12239e6e750874a25f27bc7"
server_key_version: "encryptionkey@0"
I20260926 16:44:46.581964 2496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:46.594925 2496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:46.597694 2496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:46.597915 2496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:46.604444 2496 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.125:39091
I20260926 16:44:46.604503 2563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.125:39091 every 8 connection(s)
I20260926 16:44:46.605350 2496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-1/data/info.pb
I20260926 16:44:46.607932 2564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:44:46.611431 2564 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46021 } has no permanent_uuid. Determining permanent_uuid...
I20260926 16:44:46.612648 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 2496
I20260926 16:44:46.612761 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-1/wal/instance
I20260926 16:44:46.612998 21553 external_mini_cluster.cc:1428] Setting key 7d0b303e8c52c960b02936c12f6dc709
WARNING: no policy specified for kudu/127.21.12.124@KRBTEST.COM; defaulting to no policy
Sep 26 16:44:46 dist-test-slave-n1hw krb5kdc[2407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441086, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.125@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.124@KRBTEST.COM" created.
I20260926 16:44:46.625458 2492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:46.613198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40692 (local address 127.21.12.126:46021)
0926 16:44:46.613365 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:46.613372 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:46.614222 (+ 850us) server_negotiation.cc:407] Connection header received
0926 16:44:46.615232 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:46.615239 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:46.615299 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:46.615404 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:46.616464 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.617368 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.618522 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.618921 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.619034 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:46.621515 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:46.621518 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:46.621520 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:46.621545 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:46.623060 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:46.623675 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:46.623682 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:46.623686 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:46.623778 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:46.624174 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:46.624176 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:46.624177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:46.624342 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:46.624414 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:46.625101 (+ 687us) server_negotiation.cc:299] Negotiation successful
0926 16:44:46.625246 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 16:44:46.626187 2566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:46.613549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:46021 (local address 127.0.0.1:40692)
0926 16:44:46.614037 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:46.614104 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:46.615015 (+ 911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:46.615531 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:46.615538 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:46.615851 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:46.616316 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:46.616328 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.617492 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.617495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:46.618404 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:46.618414 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.619059 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.619062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:46.619210 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:46.619804 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:46.619819 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:46.621346 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:46.623188 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:46.623193 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:46.623204 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:46.623551 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:46.623882 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:46.623885 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:46.623887 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:46.624072 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:46.624441 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:46.624450 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:46.624809 (+ 359us) client_negotiation.cc:769] Sending connection context
0926 16:44:46.625031 (+ 222us) client_negotiation.cc:240] Negotiation successful
0926 16:44:46.625340 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":106,"threads_started":1}
I20260926 16:44:46.627593 2564 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.125" port: 39091 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:44:46 dist-test-slave-n1hw krb5kdc[2407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441086, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.125@KRBTEST.COM for kudu/127.21.12.125@KRBTEST.COM
I20260926 16:44:46.637188 2566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:46.628117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.125:39091 (local address 127.0.0.1:52942)
0926 16:44:46.628247 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:46.628261 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:46.628327 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:46.628829 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:46.628832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:46.629001 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:46.629081 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:46.629086 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.630066 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.630072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:46.630609 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:46.630616 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.631204 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.631208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:46.631295 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:46.631714 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:46.631725 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:46.633885 (+ 2160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:46.635679 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:46.635685 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:46.635689 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:46.635917 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:46.636176 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:46.636180 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:46.636182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:46.636259 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:46.636703 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:46.636706 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:46.636785 (+ 79us) client_negotiation.cc:769] Sending connection context
0926 16:44:46.636897 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 16:44:46.637021 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 16:44:46.637317 2569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:46.628218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52942 (local address 127.21.12.125:39091)
0926 16:44:46.628526 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:46.628535 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:46.628551 (+ 16us) server_negotiation.cc:407] Connection header received
0926 16:44:46.628581 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:46.628585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:46.628719 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:46.628851 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:46.629192 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.629957 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.630734 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.631089 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.631209 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:46.634015 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:46.634029 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:46.634040 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:46.634058 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:46.635579 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:46.636009 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:46.636019 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:46.636021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:46.636088 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:46.636368 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:46.636371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:46.636373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:46.636604 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:46.636740 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:46.637019 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 16:44:46.637143 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":140,"threads_started":1}
I20260926 16:44:46.639179 2564 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.124" port: 38779 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.124.keytab.
Entry for principal kudu/127.21.12.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.124.keytab.
I20260926 16:44:46.639897 2566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:46.639591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:38779 (local address 127.0.0.1:40562)
0926 16:44:46.639721 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:46.639808 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.124:38779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 16:44:46.640098 2514 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.12.124:38779: connect: Connection refused (error 111)
I20260926 16:44:46.640548 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.124:38779
--webserver_interface=127.21.12.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.124.keytab
--principal=kudu/127.21.12.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:44591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:46021,127.21.12.125:39091,127.21.12.124:38779
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:44:46.640869 2566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:46.640698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:38779 (local address 127.0.0.1:40574)
0926 16:44:46.640781 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:46.640826 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.124:38779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260926 16:44:46.641014 2564 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.124:38779: Network error: Client connection negotiation failed: client connection to 127.21.12.124:38779: connect: Connection refused (error 111)
I20260926 16:44:46.649603 2488 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.125" port: 39091 } attempt: 3
Sep 26 16:44:46 dist-test-slave-n1hw krb5kdc[2407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441086, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for kudu/127.21.12.125@KRBTEST.COM
I20260926 16:44:46.659404 2491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:46.650107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.125:39091 (local address 127.0.0.1:52954)
0926 16:44:46.650242 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:46.650257 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:46.650357 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:46.650624 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:46.650627 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:46.650792 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:46.650898 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:46.650904 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.651620 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.651624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:46.652694 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:46.652709 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.653431 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.653436 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:46.653568 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:46.654173 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:46.654188 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:46.655676 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:46.657527 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:46.657530 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:46.657532 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:46.657863 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:46.658215 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:46.658219 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:46.658222 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:46.658341 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:46.658885 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:46.658889 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:46.658993 (+ 104us) client_negotiation.cc:769] Sending connection context
0926 16:44:46.659114 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 16:44:46.659242 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 16:44:46.659497 2569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:46.650180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52954 (local address 127.21.12.125:39091)
0926 16:44:46.650285 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:46.650291 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:46.650306 (+ 15us) server_negotiation.cc:407] Connection header received
0926 16:44:46.650458 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:46.650461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:46.650503 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:46.650580 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:46.651010 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.651510 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.652855 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.653291 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.653401 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:46.655800 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:46.655803 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:46.655805 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:46.655825 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:46.657392 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:46.657994 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:46.657999 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:46.658000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:46.658095 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:46.658487 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:46.658492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:46.658495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:46.658768 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:46.658882 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:46.659183 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 16:44:46.659319 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 16:44:46.660713 2488 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.124" port: 38779 } has no permanent_uuid. Determining permanent_uuid...
I20260926 16:44:46.661537 2491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:46.661231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:38779 (local address 127.0.0.1:40588)
0926 16:44:46.661387 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:46.661452 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.124:38779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 16:44:46.662531 2491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:46.662051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:38779 (local address 127.0.0.1:40600)
0926 16:44:46.662393 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:46.662447 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.124:38779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":98,"threads_started":1}
W20260926 16:44:46.662662 2488 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.124:38779: Network error: Client connection negotiation failed: client connection to 127.21.12.124:38779: connect: Connection refused (error 111)
I20260926 16:44:46.690243 2564 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.124" port: 38779 } attempt: 1
I20260926 16:44:46.691061 2566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:46.690783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:38779 (local address 127.0.0.1:40606)
0926 16:44:46.690925 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:46.690987 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.124:38779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 16:44:46.691756 2566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:46.691537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:38779 (local address 127.0.0.1:40618)
0926 16:44:46.691641 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:46.691699 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.124:38779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 16:44:46.691879 2564 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.124:38779: Network error: Client connection negotiation failed: client connection to 127.21.12.124:38779: connect: Connection refused (error 111)
I20260926 16:44:46.693800 2488 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.124" port: 38779 } attempt: 1
I20260926 16:44:46.694446 2572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:46.694177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:38779 (local address 127.0.0.1:40626)
0926 16:44:46.694296 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:46.694356 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.124:38779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 16:44:46.695111 2572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:46.694906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:38779 (local address 127.0.0.1:40636)
0926 16:44:46.695006 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:46.695049 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.124:38779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 16:44:46.695252 2488 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.124:38779: Network error: Client connection negotiation failed: client connection to 127.21.12.124:38779: connect: Connection refused (error 111)
W20260926 16:44:46.754068 2571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:46.754347 2571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:46.754424 2571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:46.758386 2571 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:44:46.758463 2571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:46.758500 2571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:46.758551 2571 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:44:46.758596 2571 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:44:46.760031 2564 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.124" port: 38779 } attempt: 2
I20260926 16:44:46.760818 2566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:46.760529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:38779 (local address 127.0.0.1:40644)
0926 16:44:46.760672 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:46.760747 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.124:38779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 16:44:46.761546 2566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:46.761344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:38779 (local address 127.0.0.1:40660)
0926 16:44:46.761447 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:46.761497 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.124:38779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 16:44:46.761673 2564 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.124:38779: Network error: Client connection negotiation failed: client connection to 127.21.12.124:38779: connect: Connection refused (error 111)
I20260926 16:44:46.764611 2571 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:44591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:46021,127.21.12.125:39091,127.21.12.124:38779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.124.keytab
--principal=kudu/127.21.12.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.124:38779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.12.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:44:46.765626 2571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:46.766721 2571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 16:44:46.770398 2488 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.124" port: 38779 } attempt: 2
I20260926 16:44:46.771148 2572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:46.770867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:38779 (local address 127.0.0.1:40674)
0926 16:44:46.771028 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:46.771081 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.124:38779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 16:44:46.771889 2572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:46.771694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:38779 (local address 127.0.0.1:40682)
0926 16:44:46.771795 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:46.771840 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.124:38779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 16:44:46.772068 2488 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.124:38779: Network error: Client connection negotiation failed: client connection to 127.21.12.124:38779: connect: Connection refused (error 111)
W20260926 16:44:46.775213 2577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:46.775218 2578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:46.775218 2580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:46.775468 2571 server_base.cc:1034] running on GCE node
I20260926 16:44:46.775997 2571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:46.777043 2571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:46.778245 2571 hybrid_clock.cc:648] HybridClock initialized: now 1790441086778210 us; error 44 us; skew 500 ppm
Sep 26 16:44:46 dist-test-slave-n1hw krb5kdc[2407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441086, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:46.781131 2571 init.cc:377] Logged in from keytab as kudu/127.21.12.124@KRBTEST.COM (short username kudu)
I20260926 16:44:46.782037 2571 webserver.cc:460] Webserver started at http://127.21.12.124:38065/ using document root <none> and password file <none>
I20260926 16:44:46.782559 2571 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:46.782616 2571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:46.782770 2571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:44:46.784721 2571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-2/data/instance:
uuid: "45084682d71044e0859cd96456a1e479"
format_stamp: "Formatted at 2026-09-26 16:44:46 on dist-test-slave-n1hw"
server_key: "167a51634451d4225ac388b5be847efe"
server_key_iv: "d0394bb59c3661f60f6630a4ea489a07"
server_key_version: "encryptionkey@0"
I20260926 16:44:46.785280 2571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-2/wal/instance:
uuid: "45084682d71044e0859cd96456a1e479"
format_stamp: "Formatted at 2026-09-26 16:44:46 on dist-test-slave-n1hw"
server_key: "167a51634451d4225ac388b5be847efe"
server_key_iv: "d0394bb59c3661f60f6630a4ea489a07"
server_key_version: "encryptionkey@0"
I20260926 16:44:46.789028 2571 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 16:44:46.791409 2587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:46.792644 2571 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 16:44:46.792753 2571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-2/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-2/wal
uuid: "45084682d71044e0859cd96456a1e479"
format_stamp: "Formatted at 2026-09-26 16:44:46 on dist-test-slave-n1hw"
server_key: "167a51634451d4225ac388b5be847efe"
server_key_iv: "d0394bb59c3661f60f6630a4ea489a07"
server_key_version: "encryptionkey@0"
I20260926 16:44:46.792834 2571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:46.809629 2571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:46.812875 2571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:46.813028 2571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:46.820061 2571 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.124:38779
I20260926 16:44:46.820094 2639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.124:38779 every 8 connection(s)
I20260926 16:44:46.821295 2571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-2/data/info.pb
I20260926 16:44:46.824046 2640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:44:46.826519 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 2571
I20260926 16:44:46.826632 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440924937351-21553-0/minicluster-data/master-2/wal/instance
I20260926 16:44:46.826884 21553 external_mini_cluster.cc:1428] Setting key 3c507b496e7bfe0870e9a29f94ae54d4
I20260926 16:44:46.828243 2640 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46021 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:44:46 dist-test-slave-n1hw krb5kdc[2407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441086, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.124@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
Sep 26 16:44:46 dist-test-slave-n1hw krb5kdc[2407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.124@KRBTEST.COM
I20260926 16:44:46.839823 2492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:46.829877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40706 (local address 127.21.12.126:46021)
0926 16:44:46.830024 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:46.830031 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:46.830473 (+ 442us) server_negotiation.cc:407] Connection header received
0926 16:44:46.830601 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:46.830606 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:46.830667 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:46.830776 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:46.831692 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.832184 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.833286 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.833504 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.833624 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:46.835978 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:46.835981 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:46.835983 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:46.836006 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:46.837567 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:46.838171 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:46.838174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:46.838175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:46.838249 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:46.838631 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:46.838634 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:46.838636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:46.838837 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:46.838949 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:46.839503 (+ 554us) server_negotiation.cc:299] Negotiation successful
0926 16:44:46.839648 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 16:44:46.840766 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:46.828150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40690 (local address 127.21.12.124:38779)
0926 16:44:46.828572 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:46.828600 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:46.828636 (+ 36us) server_negotiation.cc:407] Connection header received
0926 16:44:46.829251 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:46.829267 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:46.829670 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:46.830036 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:46.831346 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.832269 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.833089 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.833459 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.833627 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:46.836390 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:46.836398 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:46.836412 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:46.836438 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:46.838319 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:46.838746 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:46.838750 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:46.838751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:46.838817 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:46.839100 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:46.839103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:46.839104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:46.839349 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:46.839524 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:46.839733 (+ 209us) server_negotiation.cc:299] Negotiation successful
0926 16:44:46.839846 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":125,"threads_started":1}
I20260926 16:44:46.840765 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:46.829820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:46021 (local address 127.0.0.1:40706)
0926 16:44:46.830327 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:46.830381 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:46.830508 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:46.830848 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:46.830853 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:46.831325 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:46.831534 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:46.831545 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.832422 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.832425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:46.833162 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:46.833171 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.833648 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.833653 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:46.833798 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:46.834434 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:46.834449 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:46.835834 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:46.837670 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:46.837679 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:46.837690 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:46.838067 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:46.838332 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:46.838336 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:46.838339 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:46.838503 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:46.838952 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:46.838957 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:46.839242 (+ 285us) client_negotiation.cc:769] Sending connection context
0926 16:44:46.839430 (+ 188us) client_negotiation.cc:240] Negotiation successful
0926 16:44:46.839746 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":81,"threads_started":1}
I20260926 16:44:46.842376 2640 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.125" port: 39091 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:44:46 dist-test-slave-n1hw krb5kdc[2407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441086, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.124@KRBTEST.COM for kudu/127.21.12.125@KRBTEST.COM
Sep 26 16:44:46 dist-test-slave-n1hw krb5kdc[2407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:44:46.850807 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:46.842845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.125:39091 (local address 127.0.0.1:52958)
0926 16:44:46.842940 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:46.842956 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:46.843030 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:46.843460 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:46.843463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:46.843628 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:46.843714 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:46.843719 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.844422 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.844428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:46.844937 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:46.844944 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.845384 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.845387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:46.845480 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:46.845972 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:46.845988 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:46.847402 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:46.849325 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:46.849328 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:46.849330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:46.849521 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:46.849849 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:46.849852 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:46.849854 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:46.849938 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:46.850311 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:46.850314 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:46.850407 (+ 93us) client_negotiation.cc:769] Sending connection context
0926 16:44:46.850508 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 16:44:46.850650 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 16:44:46.850811 2569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:46.843144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52958 (local address 127.21.12.125:39091)
0926 16:44:46.843260 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:46.843265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:46.843277 (+ 12us) server_negotiation.cc:407] Connection header received
0926 16:44:46.843321 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:46.843323 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:46.843360 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:46.843448 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:46.843826 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.844321 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.845059 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.845284 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.845387 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:46.847530 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:46.847534 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:46.847537 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:46.847564 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:46.849210 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:46.849633 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:46.849637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:46.849639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:46.849719 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:46.850024 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:46.850029 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:46.850031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:46.850216 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:46.850320 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:46.850561 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 16:44:46.850676 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 16:44:46.851553 2640 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.124" port: 38779 } has no permanent_uuid. Determining permanent_uuid...
I20260926 16:44:46.851884 2492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:46.843200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40708 (local address 127.21.12.126:46021)
0926 16:44:46.843305 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:46.843309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:46.843378 (+ 69us) server_negotiation.cc:407] Connection header received
0926 16:44:46.843512 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:46.843515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:46.843557 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:46.843642 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:46.844188 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.844658 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.845391 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.845604 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.845700 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:46.847978 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:46.847986 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:46.847989 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:46.848009 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:46.849684 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:46.850224 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:46.850228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:46.850230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:46.850314 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:46.850621 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:46.850627 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:46.850631 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:46.850809 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:46.851076 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:46.851560 (+ 484us) server_negotiation.cc:299] Negotiation successful
0926 16:44:46.851683 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 16:44:46.852831 2564 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.124" port: 38779 } attempt: 3
Sep 26 16:44:46 dist-test-slave-n1hw krb5kdc[2407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441086, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.124@KRBTEST.COM for kudu/127.21.12.124@KRBTEST.COM
Sep 26 16:44:46 dist-test-slave-n1hw krb5kdc[2407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.125@KRBTEST.COM
Sep 26 16:44:46 dist-test-slave-n1hw krb5kdc[2407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441086, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.125@KRBTEST.COM for kudu/127.21.12.124@KRBTEST.COM
I20260926 16:44:46.862023 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:46.852073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:38779 (local address 127.0.0.1:40694)
0926 16:44:46.852181 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:46.852196 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:46.852263 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:46.852547 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:46.852549 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:46.852702 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:46.852758 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:46.852763 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.853797 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.853805 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:46.854756 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:46.854768 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.855380 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.855383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:46.855476 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:46.855913 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:46.855926 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:46.857402 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:46.859265 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:46.859271 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:46.859274 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:46.859842 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:46.860144 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:46.860146 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:46.860148 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:46.860256 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:46.860790 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:46.860794 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:46.860893 (+ 99us) client_negotiation.cc:769] Sending connection context
0926 16:44:46.861023 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 16:44:46.861815 (+ 792us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":6}
I20260926 16:44:46.862156 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:46.852284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40694 (local address 127.21.12.124:38779)
0926 16:44:46.852378 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:46.852382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:46.852399 (+ 17us) server_negotiation.cc:407] Connection header received
0926 16:44:46.852421 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:46.852423 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:46.852450 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:46.852525 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:46.852886 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.853656 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.854895 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.855278 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.855380 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:46.857564 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:46.857572 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:46.857576 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:46.857600 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:46.859159 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:46.859940 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:46.859946 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:46.859949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:46.860057 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:46.860372 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:46.860378 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:46.860381 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:46.860641 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:46.860750 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:46.861189 (+ 439us) server_negotiation.cc:299] Negotiation successful
0926 16:44:46.861920 (+ 731us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":579,"server-negotiator.queue_time_us":36}
I20260926 16:44:46.864033 2569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:46.853048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52974 (local address 127.21.12.125:39091)
0926 16:44:46.853153 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:46.853157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:46.853270 (+ 113us) server_negotiation.cc:407] Connection header received
0926 16:44:46.853594 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:46.853598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:46.853631 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:46.853678 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:46.854518 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.855094 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.856185 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.856647 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.856730 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:46.859796 (+ 3066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:46.859802 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:46.859807 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:46.859833 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:46.861616 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:46.862499 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:46.862503 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:46.862508 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:46.862596 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:46.863086 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:46.863089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:46.863091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:46.863320 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:46.863646 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:46.863740 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 16:44:46.863869 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":77,"server-negotiator.queue_time_us":28}
I20260926 16:44:46.864609 2566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:46.853253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:38779 (local address 127.0.0.1:40708)
0926 16:44:46.853330 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:46.853341 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:46.853398 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:46.853877 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:46.853881 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:46.854063 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:46.854171 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:46.854180 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.855619 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.855623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:46.856566 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:46.856577 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.857293 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.857297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:46.857401 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:46.857887 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:46.857904 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:46.859575 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:46.862742 (+ 3167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:46.862746 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:46.862748 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:46.862972 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:46.863272 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:46.863276 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:46.863278 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:46.863366 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:46.863865 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:46.863869 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:46.863954 (+ 85us) client_negotiation.cc:769] Sending connection context
0926 16:44:46.864065 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 16:44:46.864447 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":57}
I20260926 16:44:46.865119 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:46.853322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40708 (local address 127.21.12.124:38779)
0926 16:44:46.853644 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:46.853651 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:46.853662 (+ 11us) server_negotiation.cc:407] Connection header received
0926 16:44:46.853691 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:46.853692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:46.853750 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:46.853869 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:46.854300 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.855485 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.856691 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.857152 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.857610 (+ 458us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:46.859738 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:46.859742 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:46.859749 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:46.859778 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:46.862626 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:46.863080 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:46.863092 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:46.863094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:46.863174 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:46.863481 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:46.863484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:46.863486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:46.863741 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:46.864767 (+ 1026us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:46.864845 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 16:44:46.864959 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":259,"thread_start_us":168,"threads_started":1}
I20260926 16:44:46.865617 2640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45084682d71044e0859cd96456a1e479: Bootstrap starting.
I20260926 16:44:46.867483 21553 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:44:46.867983 2564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 957dca54515a4597892043c904a5fcad: Bootstrap starting.
I20260926 16:44:46.868595 2640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45084682d71044e0859cd96456a1e479: Neither blocks nor log segments found. Creating new log.
I20260926 16:44:46.869441 2640 log.cc:824] T 00000000000000000000000000000000 P 45084682d71044e0859cd96456a1e479: Log is configured to *not* fsync() on all Append() calls
I20260926 16:44:46.870647 2564 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 957dca54515a4597892043c904a5fcad: Neither blocks nor log segments found. Creating new log.
I20260926 16:44:46.871235 2488 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.124" port: 38779 } attempt: 3
I20260926 16:44:46.871553 2640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45084682d71044e0859cd96456a1e479: No bootstrap required, opened a new log
I20260926 16:44:46.871569 2564 log.cc:824] T 00000000000000000000000000000000 P 957dca54515a4597892043c904a5fcad: Log is configured to *not* fsync() on all Append() calls
I20260926 16:44:46.873546 2564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 957dca54515a4597892043c904a5fcad: No bootstrap required, opened a new log
I20260926 16:44:46.875536 2640 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45084682d71044e0859cd96456a1e479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db1f8e5fc714522af7aabbc0545a17c" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46021 } } peers { permanent_uuid: "957dca54515a4597892043c904a5fcad" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 39091 } } peers { permanent_uuid: "45084682d71044e0859cd96456a1e479" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 38779 } }
I20260926 16:44:46.875766 2640 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45084682d71044e0859cd96456a1e479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:44:46.875813 2640 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45084682d71044e0859cd96456a1e479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45084682d71044e0859cd96456a1e479, State: Initialized, Role: FOLLOWER
I20260926 16:44:46.876299 2640 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45084682d71044e0859cd96456a1e479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db1f8e5fc714522af7aabbc0545a17c" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46021 } } peers { permanent_uuid: "957dca54515a4597892043c904a5fcad" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 39091 } } peers { permanent_uuid: "45084682d71044e0859cd96456a1e479" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 38779 } }
I20260926 16:44:46.877350 2564 raft_consensus.cc:348] T 00000000000000000000000000000000 P 957dca54515a4597892043c904a5fcad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db1f8e5fc714522af7aabbc0545a17c" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46021 } } peers { permanent_uuid: "957dca54515a4597892043c904a5fcad" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 39091 } } peers { permanent_uuid: "45084682d71044e0859cd96456a1e479" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 38779 } }
I20260926 16:44:46.877529 2564 raft_consensus.cc:374] T 00000000000000000000000000000000 P 957dca54515a4597892043c904a5fcad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Sep 26 16:44:46 dist-test-slave-n1hw krb5kdc[2407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441086, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for kudu/127.21.12.124@KRBTEST.COM
I20260926 16:44:46.877573 2564 raft_consensus.cc:729] T 00000000000000000000000000000000 P 957dca54515a4597892043c904a5fcad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 957dca54515a4597892043c904a5fcad, State: Initialized, Role: FOLLOWER
I20260926 16:44:46.877574 2640 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45084682d71044e0859cd96456a1e479 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:44:46.878265 2564 consensus_queue.cc:260] T 00000000000000000000000000000000 P 957dca54515a4597892043c904a5fcad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db1f8e5fc714522af7aabbc0545a17c" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46021 } } peers { permanent_uuid: "957dca54515a4597892043c904a5fcad" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 39091 } } peers { permanent_uuid: "45084682d71044e0859cd96456a1e479" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 38779 } }
I20260926 16:44:46.879662 2564 sys_catalog.cc:564] T 00000000000000000000000000000000 P 957dca54515a4597892043c904a5fcad [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:44:46.882022 2569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:46.869674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52978 (local address 127.21.12.125:39091)
0926 16:44:46.869987 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:46.869992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:46.870715 (+ 723us) server_negotiation.cc:407] Connection header received
0926 16:44:46.870901 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:46.870904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:46.870956 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:46.871042 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:46.872642 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.873514 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.874845 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.875219 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.875305 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:46.877499 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:46.877503 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:46.877506 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:46.877533 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:46.879670 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:46.880501 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:46.880505 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:46.880507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:46.880596 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:46.880939 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:46.880943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:46.880945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:46.881172 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:46.881292 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:46.881699 (+ 407us) server_negotiation.cc:299] Negotiation successful
0926 16:44:46.881845 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237}
W20260926 16:44:46.883131 2667 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 45084682d71044e0859cd96456a1e479: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 16:44:46.883209 2667 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 16:44:46.883486 2655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45084682d71044e0859cd96456a1e479 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db1f8e5fc714522af7aabbc0545a17c" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46021 } } peers { permanent_uuid: "957dca54515a4597892043c904a5fcad" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 39091 } } peers { permanent_uuid: "45084682d71044e0859cd96456a1e479" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 38779 } } }
I20260926 16:44:46.883610 2655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45084682d71044e0859cd96456a1e479 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 16:44:46.885335 2680 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 957dca54515a4597892043c904a5fcad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 16:44:46.885457 2680 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 16:44:46.885746 2661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 957dca54515a4597892043c904a5fcad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db1f8e5fc714522af7aabbc0545a17c" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46021 } } peers { permanent_uuid: "957dca54515a4597892043c904a5fcad" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 39091 } } peers { permanent_uuid: "45084682d71044e0859cd96456a1e479" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 38779 } } }
I20260926 16:44:46.886222 2661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 957dca54515a4597892043c904a5fcad [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:44:46.886301 2496 master_runner.cc:186] Error getting master registration for 127.21.12.126:46021: OK, instance_id { permanent_uuid: "1db1f8e5fc714522af7aabbc0545a17c" instance_seqno: 1790441086408364 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 16:44:46.891209 2496 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 16:44:46.882404 2571 master_runner.cc:186] Error getting master registration for 127.21.12.126:46021: OK, instance_id { permanent_uuid: "1db1f8e5fc714522af7aabbc0545a17c" instance_seqno: 1790441086408364 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 16:44:46.891880 2571 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 16:44:46.893635 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:46.870235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40710 (local address 127.21.12.124:38779)
0926 16:44:46.870649 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:46.870655 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:46.883601 (+ 12946us) server_negotiation.cc:407] Connection header received
0926 16:44:46.883768 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:46.883772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:46.883814 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:46.883864 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:46.884830 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.885680 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.887008 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.887422 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.887519 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:46.889464 (+ 1945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:46.889468 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:46.889471 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:46.889500 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:46.891494 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:46.892061 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:46.892068 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:46.892073 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:46.892170 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:46.892515 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:46.892519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:46.892521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:46.892755 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:46.892889 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:46.893305 (+ 416us) server_negotiation.cc:299] Negotiation successful
0926 16:44:46.893439 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340}
I20260926 16:44:46.894847 2572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:46.871611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:38779 (local address 127.0.0.1:40726)
0926 16:44:46.871716 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:46.871734 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:46.871817 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:46.872294 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:46.872298 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:46.872518 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:46.872670 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:46.872694 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.873792 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.873795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:46.874767 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:46.874781 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.875474 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.875477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:46.875586 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:46.876241 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:46.876259 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:46.878020 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:46.892969 (+ 14949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:46.892972 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:46.892975 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:46.893281 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:46.893611 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:46.893615 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:46.893617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:46.893715 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:46.894274 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:46.894278 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:46.894387 (+ 109us) client_negotiation.cc:769] Sending connection context
0926 16:44:46.894494 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 16:44:46.894675 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 16:44:46.895360 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:46.871830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40726 (local address 127.21.12.124:38779)
0926 16:44:46.871964 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:46.871969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:46.871983 (+ 14us) server_negotiation.cc:407] Connection header received
0926 16:44:46.872022 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:46.872028 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:46.872080 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:46.872164 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:46.872821 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.873630 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.874899 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.875268 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.875357 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:46.878185 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:46.878193 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:46.878196 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:46.878224 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:46.892798 (+ 14574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:46.893396 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:46.893403 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:46.893408 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:46.893484 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:46.893899 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:46.893903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:46.893905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:46.894121 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:46.894224 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:46.895099 (+ 875us) server_negotiation.cc:299] Negotiation successful
0926 16:44:46.895211 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 16:44:46.899250 2488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1db1f8e5fc714522af7aabbc0545a17c: Bootstrap starting.
I20260926 16:44:46.903180 2488 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1db1f8e5fc714522af7aabbc0545a17c: Neither blocks nor log segments found. Creating new log.
I20260926 16:44:46.903488 2492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:46.870090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40724 (local address 127.21.12.126:46021)
0926 16:44:46.870227 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:46.870232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:46.870480 (+ 248us) server_negotiation.cc:407] Connection header received
0926 16:44:46.871033 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:46.871036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:46.871083 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:46.871139 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:46.878767 (+ 7628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.879671 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.897193 (+ 17522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.897547 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.897638 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:46.899545 (+ 1907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:46.899550 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:46.899552 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:46.899577 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:46.901415 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:46.902028 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:46.902033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:46.902034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:46.902123 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:46.902443 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:46.902447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:46.902448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:46.902676 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:46.902807 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:46.903205 (+ 398us) server_negotiation.cc:299] Negotiation successful
0926 16:44:46.903330 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 16:44:46.904567 2488 log.cc:824] T 00000000000000000000000000000000 P 1db1f8e5fc714522af7aabbc0545a17c: Log is configured to *not* fsync() on all Append() calls
I20260926 16:44:46.906636 2488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1db1f8e5fc714522af7aabbc0545a17c: No bootstrap required, opened a new log
I20260926 16:44:46.911562 2488 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1db1f8e5fc714522af7aabbc0545a17c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db1f8e5fc714522af7aabbc0545a17c" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46021 } } peers { permanent_uuid: "957dca54515a4597892043c904a5fcad" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 39091 } } peers { permanent_uuid: "45084682d71044e0859cd96456a1e479" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 38779 } }
I20260926 16:44:46.911831 2488 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1db1f8e5fc714522af7aabbc0545a17c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:44:46.911890 2488 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1db1f8e5fc714522af7aabbc0545a17c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1db1f8e5fc714522af7aabbc0545a17c, State: Initialized, Role: FOLLOWER
I20260926 16:44:46.912374 2488 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1db1f8e5fc714522af7aabbc0545a17c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db1f8e5fc714522af7aabbc0545a17c" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46021 } } peers { permanent_uuid: "957dca54515a4597892043c904a5fcad" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 39091 } } peers { permanent_uuid: "45084682d71044e0859cd96456a1e479" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 38779 } }
I20260926 16:44:46.913220 2683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1db1f8e5fc714522af7aabbc0545a17c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db1f8e5fc714522af7aabbc0545a17c" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46021 } } peers { permanent_uuid: "957dca54515a4597892043c904a5fcad" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 39091 } } peers { permanent_uuid: "45084682d71044e0859cd96456a1e479" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 38779 } } }
I20260926 16:44:46.913408 2683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1db1f8e5fc714522af7aabbc0545a17c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:44:46.914130 2488 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1db1f8e5fc714522af7aabbc0545a17c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:44:46.922950 2420 master_runner.cc:263] No leader master found from master 1db1f8e5fc714522af7aabbc0545a17c
I20260926 16:44:46.923023 2420 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 16:44:46.923596 2694 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1db1f8e5fc714522af7aabbc0545a17c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 16:44:46.923740 2694 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260926 16:44:47.064332 2593 tablet.cc:2369] T 00000000000000000000000000000000 P 45084682d71044e0859cd96456a1e479: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:44:47.099311 2517 tablet.cc:2369] T 00000000000000000000000000000000 P 957dca54515a4597892043c904a5fcad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:44:47.155058 2441 tablet.cc:2369] T 00000000000000000000000000000000 P 1db1f8e5fc714522af7aabbc0545a17c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:44:47.209102 2661 raft_consensus.cc:482] T 00000000000000000000000000000000 P 957dca54515a4597892043c904a5fcad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:44:47.209285 2661 raft_consensus.cc:504] T 00000000000000000000000000000000 P 957dca54515a4597892043c904a5fcad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db1f8e5fc714522af7aabbc0545a17c" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46021 } } peers { permanent_uuid: "957dca54515a4597892043c904a5fcad" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 39091 } } peers { permanent_uuid: "45084682d71044e0859cd96456a1e479" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 38779 } }
I20260926 16:44:47.210317 2661 leader_election.cc:288] T 00000000000000000000000000000000 P 957dca54515a4597892043c904a5fcad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1db1f8e5fc714522af7aabbc0545a17c (127.21.12.126:46021), 45084682d71044e0859cd96456a1e479 (127.21.12.124:38779)
I20260926 16:44:47.210853 2614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "957dca54515a4597892043c904a5fcad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45084682d71044e0859cd96456a1e479" is_pre_election: true
I20260926 16:44:47.210856 2462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "957dca54515a4597892043c904a5fcad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1db1f8e5fc714522af7aabbc0545a17c" is_pre_election: true
I20260926 16:44:47.211179 2614 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 45084682d71044e0859cd96456a1e479 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 957dca54515a4597892043c904a5fcad in term 0.
I20260926 16:44:47.211179 2462 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1db1f8e5fc714522af7aabbc0545a17c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 957dca54515a4597892043c904a5fcad in term 0.
I20260926 16:44:47.211627 2514 leader_election.cc:302] T 00000000000000000000000000000000 P 957dca54515a4597892043c904a5fcad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1db1f8e5fc714522af7aabbc0545a17c, 45084682d71044e0859cd96456a1e479, 957dca54515a4597892043c904a5fcad; no voters:
I20260926 16:44:47.211895 2661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 957dca54515a4597892043c904a5fcad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:44:47.211978 2661 raft_consensus.cc:482] T 00000000000000000000000000000000 P 957dca54515a4597892043c904a5fcad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:44:47.212030 2661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 957dca54515a4597892043c904a5fcad [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:44:47.213068 2661 raft_consensus.cc:504] T 00000000000000000000000000000000 P 957dca54515a4597892043c904a5fcad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db1f8e5fc714522af7aabbc0545a17c" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46021 } } peers { permanent_uuid: "957dca54515a4597892043c904a5fcad" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 39091 } } peers { permanent_uuid: "45084682d71044e0859cd96456a1e479" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 38779 } }
I20260926 16:44:47.213367 2661 leader_election.cc:288] T 00000000000000000000000000000000 P 957dca54515a4597892043c904a5fcad [CANDIDATE]: Term 1 election: Requested vote from peers 1db1f8e5fc714522af7aabbc0545a17c (127.21.12.126:46021), 45084682d71044e0859cd96456a1e479 (127.21.12.124:38779)
I20260926 16:44:47.213671 2462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "957dca54515a4597892043c904a5fcad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1db1f8e5fc714522af7aabbc0545a17c"
I20260926 16:44:47.213711 2614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "957dca54515a4597892043c904a5fcad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45084682d71044e0859cd96456a1e479"
I20260926 16:44:47.213837 2462 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1db1f8e5fc714522af7aabbc0545a17c [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:44:47.213867 2614 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45084682d71044e0859cd96456a1e479 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:44:47.214986 2614 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 45084682d71044e0859cd96456a1e479 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 957dca54515a4597892043c904a5fcad in term 1.
I20260926 16:44:47.215035 2462 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1db1f8e5fc714522af7aabbc0545a17c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 957dca54515a4597892043c904a5fcad in term 1.
I20260926 16:44:47.215261 2514 leader_election.cc:302] T 00000000000000000000000000000000 P 957dca54515a4597892043c904a5fcad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45084682d71044e0859cd96456a1e479, 957dca54515a4597892043c904a5fcad; no voters:
I20260926 16:44:47.215426 2661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 957dca54515a4597892043c904a5fcad [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:44:47.215619 2661 raft_consensus.cc:686] T 00000000000000000000000000000000 P 957dca54515a4597892043c904a5fcad [term 1 LEADER]: Becoming Leader. State: Replica: 957dca54515a4597892043c904a5fcad, State: Running, Role: LEADER
I20260926 16:44:47.215939 2661 consensus_queue.cc:237] T 00000000000000000000000000000000 P 957dca54515a4597892043c904a5fcad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db1f8e5fc714522af7aabbc0545a17c" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46021 } } peers { permanent_uuid: "957dca54515a4597892043c904a5fcad" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 39091 } } peers { permanent_uuid: "45084682d71044e0859cd96456a1e479" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 38779 } }
I20260926 16:44:47.217816 2696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 957dca54515a4597892043c904a5fcad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 957dca54515a4597892043c904a5fcad. Latest consensus state: current_term: 1 leader_uuid: "957dca54515a4597892043c904a5fcad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db1f8e5fc714522af7aabbc0545a17c" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46021 } } peers { permanent_uuid: "957dca54515a4597892043c904a5fcad" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 39091 } } peers { permanent_uuid: "45084682d71044e0859cd96456a1e479" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 38779 } } }
I20260926 16:44:47.217933 2696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 957dca54515a4597892043c904a5fcad [sys.catalog]: This master's current role is: LEADER
I20260926 16:44:47.218286 2698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:44:47.219139 2698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:44:47.223609 2462 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1db1f8e5fc714522af7aabbc0545a17c [term 1 FOLLOWER]: Refusing update from remote peer 957dca54515a4597892043c904a5fcad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:44:47.223646 2614 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 45084682d71044e0859cd96456a1e479 [term 1 FOLLOWER]: Refusing update from remote peer 957dca54515a4597892043c904a5fcad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:44:47.224208 2696 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 957dca54515a4597892043c904a5fcad [LEADER]: Connected to new peer: Peer: permanent_uuid: "1db1f8e5fc714522af7aabbc0545a17c" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:44:47.224396 2661 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 957dca54515a4597892043c904a5fcad [LEADER]: Connected to new peer: Peer: permanent_uuid: "45084682d71044e0859cd96456a1e479" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 38779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:44:47.227854 2683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1db1f8e5fc714522af7aabbc0545a17c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 957dca54515a4597892043c904a5fcad. Latest consensus state: current_term: 1 leader_uuid: "957dca54515a4597892043c904a5fcad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db1f8e5fc714522af7aabbc0545a17c" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46021 } } peers { permanent_uuid: "957dca54515a4597892043c904a5fcad" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 39091 } } peers { permanent_uuid: "45084682d71044e0859cd96456a1e479" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 38779 } } }
I20260926 16:44:47.227967 2683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1db1f8e5fc714522af7aabbc0545a17c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:44:47.228048 2655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45084682d71044e0859cd96456a1e479 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 957dca54515a4597892043c904a5fcad. Latest consensus state: current_term: 1 leader_uuid: "957dca54515a4597892043c904a5fcad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db1f8e5fc714522af7aabbc0545a17c" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46021 } } peers { permanent_uuid: "957dca54515a4597892043c904a5fcad" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 39091 } } peers { permanent_uuid: "45084682d71044e0859cd96456a1e479" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 38779 } } }
I20260926 16:44:47.228157 2655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45084682d71044e0859cd96456a1e479 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:44:47.230535 2696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 957dca54515a4597892043c904a5fcad [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "957dca54515a4597892043c904a5fcad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db1f8e5fc714522af7aabbc0545a17c" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46021 } } peers { permanent_uuid: "957dca54515a4597892043c904a5fcad" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 39091 } } peers { permanent_uuid: "45084682d71044e0859cd96456a1e479" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 38779 } } }
I20260926 16:44:47.230639 2696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 957dca54515a4597892043c904a5fcad [sys.catalog]: This master's current role is: LEADER
I20260926 16:44:47.230782 2696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 957dca54515a4597892043c904a5fcad [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "957dca54515a4597892043c904a5fcad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db1f8e5fc714522af7aabbc0545a17c" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46021 } } peers { permanent_uuid: "957dca54515a4597892043c904a5fcad" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 39091 } } peers { permanent_uuid: "45084682d71044e0859cd96456a1e479" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 38779 } } }
I20260926 16:44:47.230854 2696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 957dca54515a4597892043c904a5fcad [sys.catalog]: This master's current role is: LEADER
I20260926 16:44:47.231071 2683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1db1f8e5fc714522af7aabbc0545a17c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "957dca54515a4597892043c904a5fcad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db1f8e5fc714522af7aabbc0545a17c" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46021 } } peers { permanent_uuid: "957dca54515a4597892043c904a5fcad" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 39091 } } peers { permanent_uuid: "45084682d71044e0859cd96456a1e479" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 38779 } } }
I20260926 16:44:47.231171 2683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1db1f8e5fc714522af7aabbc0545a17c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:44:47.232079 2698 catalog_manager.cc:1347] Generated new cluster ID: 6e77fa0f45534b63be3fe3d4d5d39b16
I20260926 16:44:47.232141 2698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:44:47.232172 2655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45084682d71044e0859cd96456a1e479 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "957dca54515a4597892043c904a5fcad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db1f8e5fc714522af7aabbc0545a17c" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46021 } } peers { permanent_uuid: "957dca54515a4597892043c904a5fcad" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 39091 } } peers { permanent_uuid: "45084682d71044e0859cd96456a1e479" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 38779 } } }
I20260926 16:44:47.232296 2655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45084682d71044e0859cd96456a1e479 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:44:47.247997 2698 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:44:47.249205 2698 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:44:47.264420 2698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 957dca54515a4597892043c904a5fcad: Generated new TSK 0
I20260926 16:44:47.265249 2698 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:44:47.884472 2667 catalog_manager.cc:1259] Loaded cluster ID: 6e77fa0f45534b63be3fe3d4d5d39b16
I20260926 16:44:47.884598 2667 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 45084682d71044e0859cd96456a1e479: loading cluster ID for follower catalog manager: success
I20260926 16:44:47.886555 2667 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 45084682d71044e0859cd96456a1e479: acquiring CA information for follower catalog manager: success
I20260926 16:44:47.887728 2667 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 45084682d71044e0859cd96456a1e479: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 16:44:47.925068 2694 catalog_manager.cc:1259] Loaded cluster ID: 6e77fa0f45534b63be3fe3d4d5d39b16
I20260926 16:44:47.925154 2694 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1db1f8e5fc714522af7aabbc0545a17c: loading cluster ID for follower catalog manager: success
I20260926 16:44:47.926923 2694 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1db1f8e5fc714522af7aabbc0545a17c: acquiring CA information for follower catalog manager: success
I20260926 16:44:47.928061 2694 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1db1f8e5fc714522af7aabbc0545a17c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 16:44:48.327862 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 2420
I20260926 16:44:48.334264 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 2496
I20260926 16:44:48.340021 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 2571
2026-09-26T16:44:48Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3268 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440924937351-21553-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:44:48 dist-test-slave-n1hw krb5kdc[2713](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:44:48 dist-test-slave-n1hw krb5kdc[2713](info): set up 2 sockets
Sep 26 16:44:48 dist-test-slave-n1hw krb5kdc[2713](info): commencing operation
krb5kdc: starting...
W20260926 16:44:49.400146 21553 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Sep 26 16:44:49 dist-test-slave-n1hw krb5kdc[2713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:44:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:44:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
I20260926 16:44:49.522107 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:44137
--webserver_interface=127.21.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:45455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:44137
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:44:49.629387 2726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:49.629679 2726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:49.629786 2726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:49.633771 2726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:44:49.633864 2726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:49.633900 2726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:49.633939 2726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:44:49.633970 2726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:44:49.639134 2726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:45455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:44137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:44137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:44:49.640182 2726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:49.641223 2726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:49.649420 2731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:49.649420 2732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:49.649544 2726 server_base.cc:1034] running on GCE node
W20260926 16:44:49.649441 2734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:49.650406 2726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:49.651424 2726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:49.652633 2726 hybrid_clock.cc:648] HybridClock initialized: now 1790441089652619 us; error 27 us; skew 500 ppm
Sep 26 16:44:49 dist-test-slave-n1hw krb5kdc[2713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441089, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:49.655392 2726 init.cc:377] Logged in from keytab as kudu/127.21.12.126@KRBTEST.COM (short username kudu)
I20260926 16:44:49.656234 2726 webserver.cc:460] Webserver started at http://127.21.12.126:40497/ using document root <none> and password file <none>
I20260926 16:44:49.656769 2726 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:49.656855 2726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:49.657069 2726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:44:49.659109 2726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "fdb70940a4cc4015868b5ad2fdb431c4"
format_stamp: "Formatted at 2026-09-26 16:44:49 on dist-test-slave-n1hw"
server_key: "71d3907b39018e4ecaf84b21b3193f8a"
server_key_iv: "31c1397bbb5642103e8b0f4a32c9dac0"
server_key_version: "encryptionkey@0"
I20260926 16:44:49.659657 2726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "fdb70940a4cc4015868b5ad2fdb431c4"
format_stamp: "Formatted at 2026-09-26 16:44:49 on dist-test-slave-n1hw"
server_key: "71d3907b39018e4ecaf84b21b3193f8a"
server_key_iv: "31c1397bbb5642103e8b0f4a32c9dac0"
server_key_version: "encryptionkey@0"
I20260926 16:44:49.663169 2726 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:44:49.665460 2741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:49.666740 2726 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:44:49.666862 2726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "fdb70940a4cc4015868b5ad2fdb431c4"
format_stamp: "Formatted at 2026-09-26 16:44:49 on dist-test-slave-n1hw"
server_key: "71d3907b39018e4ecaf84b21b3193f8a"
server_key_iv: "31c1397bbb5642103e8b0f4a32c9dac0"
server_key_version: "encryptionkey@0"
I20260926 16:44:49.666968 2726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:49.680186 2726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:49.683139 2726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:49.683326 2726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:49.690677 2726 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.126:44137
I20260926 16:44:49.690718 2794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.126:44137 every 8 connection(s)
I20260926 16:44:49.691619 2726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:44:49.694557 2795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:44:49.697952 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 2726
I20260926 16:44:49.698041 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:44:49.698289 21553 external_mini_cluster.cc:1428] Setting key 5bf9ba51132ba464e0d2610b993315a0
I20260926 16:44:49.700532 2795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdb70940a4cc4015868b5ad2fdb431c4: Bootstrap starting.
I20260926 16:44:49.703186 2795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdb70940a4cc4015868b5ad2fdb431c4: Neither blocks nor log segments found. Creating new log.
I20260926 16:44:49.704145 2795 log.cc:824] T 00000000000000000000000000000000 P fdb70940a4cc4015868b5ad2fdb431c4: Log is configured to *not* fsync() on all Append() calls
I20260926 16:44:49.706535 2795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdb70940a4cc4015868b5ad2fdb431c4: No bootstrap required, opened a new log
Sep 26 16:44:49 dist-test-slave-n1hw krb5kdc[2713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:44:49.709873 2795 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdb70940a4cc4015868b5ad2fdb431c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb70940a4cc4015868b5ad2fdb431c4" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 44137 } }
I20260926 16:44:49.710094 2795 raft_consensus.cc:374] T 00000000000000000000000000000000 P fdb70940a4cc4015868b5ad2fdb431c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:44:49.710191 2795 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdb70940a4cc4015868b5ad2fdb431c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdb70940a4cc4015868b5ad2fdb431c4, State: Initialized, Role: FOLLOWER
I20260926 16:44:49.710681 2795 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdb70940a4cc4015868b5ad2fdb431c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb70940a4cc4015868b5ad2fdb431c4" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 44137 } }
I20260926 16:44:49.710839 2795 raft_consensus.cc:388] T 00000000000000000000000000000000 P fdb70940a4cc4015868b5ad2fdb431c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:44:49.710928 2795 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdb70940a4cc4015868b5ad2fdb431c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:44:49.711048 2795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdb70940a4cc4015868b5ad2fdb431c4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:44:49.711946 2795 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdb70940a4cc4015868b5ad2fdb431c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb70940a4cc4015868b5ad2fdb431c4" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 44137 } }
I20260926 16:44:49.712301 2795 leader_election.cc:302] T 00000000000000000000000000000000 P fdb70940a4cc4015868b5ad2fdb431c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdb70940a4cc4015868b5ad2fdb431c4; no voters:
I20260926 16:44:49.712702 2795 leader_election.cc:288] T 00000000000000000000000000000000 P fdb70940a4cc4015868b5ad2fdb431c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:44:49.712795 2800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdb70940a4cc4015868b5ad2fdb431c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:44:49.713014 2800 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdb70940a4cc4015868b5ad2fdb431c4 [term 1 LEADER]: Becoming Leader. State: Replica: fdb70940a4cc4015868b5ad2fdb431c4, State: Running, Role: LEADER
I20260926 16:44:49.713498 2800 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdb70940a4cc4015868b5ad2fdb431c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb70940a4cc4015868b5ad2fdb431c4" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 44137 } }
I20260926 16:44:49.713843 2798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:49.699647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48052 (local address 127.21.12.126:44137)
0926 16:44:49.700035 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:49.700051 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:49.700089 (+ 38us) server_negotiation.cc:407] Connection header received
0926 16:44:49.700731 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:49.700749 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:49.701068 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:49.701328 (+ 260us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:49.702333 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:49.703348 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:49.704601 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:49.705190 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:49.705310 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:49.708186 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:49.708190 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:49.708202 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:49.708227 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:49.710622 (+ 2395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:49.711126 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:49.711132 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:49.711154 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:49.711280 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:49.711562 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:49.711568 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:49.711572 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:49.711943 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:49.712149 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:49.712386 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 16:44:49.712719 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":121,"threads_started":1}
I20260926 16:44:49.714125 2795 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdb70940a4cc4015868b5ad2fdb431c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:44:49.714740 2800 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdb70940a4cc4015868b5ad2fdb431c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdb70940a4cc4015868b5ad2fdb431c4. Latest consensus state: current_term: 1 leader_uuid: "fdb70940a4cc4015868b5ad2fdb431c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb70940a4cc4015868b5ad2fdb431c4" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 44137 } } }
I20260926 16:44:49.714917 2800 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdb70940a4cc4015868b5ad2fdb431c4 [sys.catalog]: This master's current role is: LEADER
I20260926 16:44:49.715215 2801 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdb70940a4cc4015868b5ad2fdb431c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fdb70940a4cc4015868b5ad2fdb431c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb70940a4cc4015868b5ad2fdb431c4" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 44137 } } }
I20260926 16:44:49.715294 2804 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:44:49.715351 2801 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdb70940a4cc4015868b5ad2fdb431c4 [sys.catalog]: This master's current role is: LEADER
I20260926 16:44:49.717121 21553 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:44:49.718449 2804 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:44:49.724040 2804 catalog_manager.cc:1347] Generated new cluster ID: 183ea213422d4625892fa9523cc30ab8
I20260926 16:44:49.724108 2804 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:44:49.732903 2804 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:44:49.734206 2804 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:44:49.744817 2804 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fdb70940a4cc4015868b5ad2fdb431c4: Generated new TSK 0
I20260926 16:44:49.745464 2804 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:44:49.773306 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 2726
2026-09-26T16:44:49Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1431 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:44:49 dist-test-slave-n1hw krb5kdc[2822](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:44:49 dist-test-slave-n1hw krb5kdc[2822](info): set up 2 sockets
Sep 26 16:44:49 dist-test-slave-n1hw krb5kdc[2822](info): commencing operation
krb5kdc: starting...
W20260926 16:44:50.812227 21553 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Sep 26 16:44:50 dist-test-slave-n1hw krb5kdc[2822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:44:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:44:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
I20260926 16:44:50.933815 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:43623
--webserver_interface=127.21.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:38015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:43623
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:44:51.040498 2835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:51.040794 2835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:51.040867 2835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:51.045403 2835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:44:51.045481 2835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:51.045518 2835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:51.045555 2835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:44:51.045598 2835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:44:51.050858 2835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:38015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:43623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:43623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:44:51.051898 2835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:51.052906 2835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:51.061288 2843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:51.061322 2835 server_base.cc:1034] running on GCE node
W20260926 16:44:51.061322 2840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:51.061288 2841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:51.062237 2835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:51.063344 2835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:51.064534 2835 hybrid_clock.cc:648] HybridClock initialized: now 1790441091064515 us; error 32 us; skew 500 ppm
Sep 26 16:44:51 dist-test-slave-n1hw krb5kdc[2822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441091, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:51.067341 2835 init.cc:377] Logged in from keytab as kudu/127.21.12.126@KRBTEST.COM (short username kudu)
I20260926 16:44:51.068249 2835 webserver.cc:460] Webserver started at http://127.21.12.126:42949/ using document root <none> and password file <none>
I20260926 16:44:51.068790 2835 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:51.068846 2835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:51.069010 2835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:44:51.070995 2835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "e3ea10b58512425eb10bf0d7a55d3e17"
format_stamp: "Formatted at 2026-09-26 16:44:51 on dist-test-slave-n1hw"
server_key: "60ea5f920d55bd0c05a5067892acc098"
server_key_iv: "4b0e4304b593a2a1d6bf2f650ffcc8d2"
server_key_version: "encryptionkey@0"
I20260926 16:44:51.071483 2835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "e3ea10b58512425eb10bf0d7a55d3e17"
format_stamp: "Formatted at 2026-09-26 16:44:51 on dist-test-slave-n1hw"
server_key: "60ea5f920d55bd0c05a5067892acc098"
server_key_iv: "4b0e4304b593a2a1d6bf2f650ffcc8d2"
server_key_version: "encryptionkey@0"
I20260926 16:44:51.074972 2835 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:44:51.077193 2850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:51.078681 2835 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 16:44:51.078783 2835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "e3ea10b58512425eb10bf0d7a55d3e17"
format_stamp: "Formatted at 2026-09-26 16:44:51 on dist-test-slave-n1hw"
server_key: "60ea5f920d55bd0c05a5067892acc098"
server_key_iv: "4b0e4304b593a2a1d6bf2f650ffcc8d2"
server_key_version: "encryptionkey@0"
I20260926 16:44:51.078922 2835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:51.116887 2835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:51.120220 2835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:51.120378 2835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:51.127194 2835 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.126:43623
I20260926 16:44:51.127470 2903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.126:43623 every 8 connection(s)
I20260926 16:44:51.128171 2835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:44:51.129727 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 2835
I20260926 16:44:51.129856 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:44:51.130133 21553 external_mini_cluster.cc:1428] Setting key 4ac075b8277f97262f8f2c52b886eab2
I20260926 16:44:51.131209 2904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:44:51.137723 2904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17: Bootstrap starting.
Sep 26 16:44:51 dist-test-slave-n1hw krb5kdc[2822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:44:51.139947 2904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17: Neither blocks nor log segments found. Creating new log.
I20260926 16:44:51.140882 2904 log.cc:824] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17: Log is configured to *not* fsync() on all Append() calls
I20260926 16:44:51.143002 2904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17: No bootstrap required, opened a new log
I20260926 16:44:51.144937 2907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:51.131478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37864 (local address 127.21.12.126:43623)
0926 16:44:51.131927 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:51.131945 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:51.131986 (+ 41us) server_negotiation.cc:407] Connection header received
0926 16:44:51.132812 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:51.132833 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:51.133230 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:51.133585 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:51.134332 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:51.135773 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:51.136621 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:51.137310 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:51.137463 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:51.139881 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:51.139889 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:51.139906 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:51.139938 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:51.141790 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:51.142195 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:51.142200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:51.142206 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:51.142335 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:51.142586 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:51.142589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:51.142591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:51.143054 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:51.143249 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:51.143552 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 16:44:51.143828 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":102,"threads_started":1}
I20260926 16:44:51.146652 2904 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ea10b58512425eb10bf0d7a55d3e17" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 43623 } }
I20260926 16:44:51.146889 2904 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:44:51.146950 2904 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3ea10b58512425eb10bf0d7a55d3e17, State: Initialized, Role: FOLLOWER
I20260926 16:44:51.147375 2904 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ea10b58512425eb10bf0d7a55d3e17" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 43623 } }
I20260926 16:44:51.147516 2904 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:44:51.147588 2904 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:44:51.147686 2904 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:44:51.148558 2904 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ea10b58512425eb10bf0d7a55d3e17" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 43623 } }
I20260926 16:44:51.148881 2904 leader_election.cc:302] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3ea10b58512425eb10bf0d7a55d3e17; no voters:
I20260926 16:44:51.149163 2904 leader_election.cc:288] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:44:51.149305 2909 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:44:51.149546 2909 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [term 1 LEADER]: Becoming Leader. State: Replica: e3ea10b58512425eb10bf0d7a55d3e17, State: Running, Role: LEADER
I20260926 16:44:51.149917 2909 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ea10b58512425eb10bf0d7a55d3e17" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 43623 } }
I20260926 16:44:51.150349 2904 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:44:51.151727 2910 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3ea10b58512425eb10bf0d7a55d3e17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ea10b58512425eb10bf0d7a55d3e17" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 43623 } } }
I20260926 16:44:51.151729 2911 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3ea10b58512425eb10bf0d7a55d3e17. Latest consensus state: current_term: 1 leader_uuid: "e3ea10b58512425eb10bf0d7a55d3e17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ea10b58512425eb10bf0d7a55d3e17" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 43623 } } }
I20260926 16:44:51.151878 2910 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [sys.catalog]: This master's current role is: LEADER
I20260926 16:44:51.151888 2911 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [sys.catalog]: This master's current role is: LEADER
I20260926 16:44:51.152252 2919 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:44:51.155131 2919 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:44:51.160317 2919 catalog_manager.cc:1347] Generated new cluster ID: 6acfdeb002a9424981a3f8091bcff7cc
I20260926 16:44:51.160377 2919 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:44:51.173921 2919 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:44:51.174782 2919 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:44:51.189215 2919 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17: Generated new TSK 0
I20260926 16:44:51.189931 2919 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab.
Entry for principal kudu/127.21.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab.
I20260926 16:44:51.220574 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.65:0
--local_ip_for_outbound_sockets=127.21.12.65
--webserver_interface=127.21.12.65
--webserver_port=0
--tserver_master_addrs=127.21.12.126:43623
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab
--principal=kudu/127.21.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:38015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:44:51.328104 2930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:51.328325 2930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:51.328363 2930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:51.332182 2930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:51.332230 2930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:51.332284 2930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.65
I20260926 16:44:51.338236 2930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:38015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab
--principal=kudu/127.21.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.65
--webserver_port=0
--tserver_master_addrs=127.21.12.126:43623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.65
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:44:51.339363 2930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:51.340319 2930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:51.349221 2938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:51.349221 2936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:51.349313 2935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:51.349637 2930 server_base.cc:1034] running on GCE node
I20260926 16:44:51.350065 2930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:51.350660 2930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:51.351927 2930 hybrid_clock.cc:648] HybridClock initialized: now 1790441091351911 us; error 36 us; skew 500 ppm
Sep 26 16:44:51 dist-test-slave-n1hw krb5kdc[2822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441091, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:51.354780 2930 init.cc:377] Logged in from keytab as kudu/127.21.12.65@KRBTEST.COM (short username kudu)
I20260926 16:44:51.355681 2930 webserver.cc:460] Webserver started at http://127.21.12.65:44711/ using document root <none> and password file <none>
I20260926 16:44:51.356220 2930 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:51.356280 2930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:51.356480 2930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:44:51.358501 2930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-0/data/instance:
uuid: "e2f1b9f64c4d4917a133f18b471fab50"
format_stamp: "Formatted at 2026-09-26 16:44:51 on dist-test-slave-n1hw"
server_key: "afaceace097be71f67d76240e609df8c"
server_key_iv: "5c7e68d2ff7d108238f7e847e8bc4e2a"
server_key_version: "encryptionkey@0"
I20260926 16:44:51.359048 2930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance:
uuid: "e2f1b9f64c4d4917a133f18b471fab50"
format_stamp: "Formatted at 2026-09-26 16:44:51 on dist-test-slave-n1hw"
server_key: "afaceace097be71f67d76240e609df8c"
server_key_iv: "5c7e68d2ff7d108238f7e847e8bc4e2a"
server_key_version: "encryptionkey@0"
I20260926 16:44:51.362641 2930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 16:44:51.364858 2945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:51.366175 2930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:44:51.366322 2930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-0/wal
uuid: "e2f1b9f64c4d4917a133f18b471fab50"
format_stamp: "Formatted at 2026-09-26 16:44:51 on dist-test-slave-n1hw"
server_key: "afaceace097be71f67d76240e609df8c"
server_key_iv: "5c7e68d2ff7d108238f7e847e8bc4e2a"
server_key_version: "encryptionkey@0"
I20260926 16:44:51.366428 2930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:51.396601 2930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:51.399523 2930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:51.399734 2930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:51.400305 2930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:44:51.401129 2930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:44:51.401197 2930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:51.401270 2930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:44:51.401321 2930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:51.410075 2930 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.65:41567
I20260926 16:44:51.410105 3059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.65:41567 every 8 connection(s)
I20260926 16:44:51.410929 2930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
I20260926 16:44:51.416453 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 2930
I20260926 16:44:51.416555 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance
I20260926 16:44:51.416824 21553 external_mini_cluster.cc:1428] Setting key 8586c0e42351cd354dfd486acc23f5a6
Sep 26 16:44:51 dist-test-slave-n1hw krb5kdc[2822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441091, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:44:51.425908 2907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:51.412593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:36357 (local address 127.21.12.126:43623)
0926 16:44:51.412727 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:51.412730 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:51.413504 (+ 774us) server_negotiation.cc:407] Connection header received
0926 16:44:51.414924 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:51.414927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:51.414978 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:51.415059 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:51.416389 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:51.417102 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:51.418504 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:51.418911 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:51.419042 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:51.421690 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:51.421694 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:51.421696 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:51.421719 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:51.423401 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:51.424103 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:51.424106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:51.424107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:51.424189 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:51.424589 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:51.424591 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:51.424592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:51.424801 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:51.424879 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:51.425575 (+ 696us) server_negotiation.cc:299] Negotiation successful
0926 16:44:51.425699 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
WARNING: no policy specified for kudu/127.21.12.66@KRBTEST.COM; defaulting to no policy
I20260926 16:44:51.427059 3062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:51.412920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.65:36357)
0926 16:44:51.413350 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:51.413391 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:51.414707 (+ 1316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:51.415258 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:51.415265 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:51.415665 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:51.416194 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:51.416217 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:51.417262 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:51.417266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:51.418384 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:51.418394 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:51.419098 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:51.419101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:51.419254 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:51.419911 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:51.419928 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:51.421495 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:51.423532 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:51.423541 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:51.423561 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:51.423957 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:51.424292 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:51.424298 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:51.424302 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:51.424488 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:51.424915 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:51.424922 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:51.425273 (+ 351us) client_negotiation.cc:769] Sending connection context
0926 16:44:51.425505 (+ 232us) client_negotiation.cc:240] Negotiation successful
0926 16:44:51.425897 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":122,"threads_started":1}
I20260926 16:44:51.428303 3060 heartbeater.cc:347] Connected to a master server at 127.21.12.126:43623
I20260926 16:44:51.428553 3060 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:51.429116 3060 heartbeater.cc:511] Master 127.21.12.126:43623 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.66@KRBTEST.COM" created.
I20260926 16:44:51.430737 2865 ts_manager.cc:194] Registered new tserver with Master: e2f1b9f64c4d4917a133f18b471fab50 (127.21.12.65:41567)
I20260926 16:44:51.431842 2865 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.65@KRBTEST.COM'} at 127.21.12.65:36357
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab.
Entry for principal kudu/127.21.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab.
I20260926 16:44:51.445258 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.66:0
--local_ip_for_outbound_sockets=127.21.12.66
--webserver_interface=127.21.12.66
--webserver_port=0
--tserver_master_addrs=127.21.12.126:43623
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab
--principal=kudu/127.21.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:38015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:44:51.554278 3065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:51.554507 3065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:51.554580 3065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:51.558602 3065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:51.558655 3065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:51.558755 3065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.66
I20260926 16:44:51.563880 3065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:38015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab
--principal=kudu/127.21.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.66
--webserver_port=0
--tserver_master_addrs=127.21.12.126:43623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.66
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:44:51.564862 3065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:51.565912 3065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 16:44:51.575364 3065 server_base.cc:1034] running on GCE node
W20260926 16:44:51.575390 3073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:51.575389 3070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:51.575343 3071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:51.575950 3065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:51.576570 3065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:51.577831 3065 hybrid_clock.cc:648] HybridClock initialized: now 1790441091577817 us; error 23 us; skew 500 ppm
Sep 26 16:44:51 dist-test-slave-n1hw krb5kdc[2822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441091, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:51.580562 3065 init.cc:377] Logged in from keytab as kudu/127.21.12.66@KRBTEST.COM (short username kudu)
I20260926 16:44:51.581382 3065 webserver.cc:460] Webserver started at http://127.21.12.66:41773/ using document root <none> and password file <none>
I20260926 16:44:51.581985 3065 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:51.582077 3065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:51.582329 3065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:44:51.584436 3065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-1/data/instance:
uuid: "9e58b0c44a5b4076b135b9a2284d2335"
format_stamp: "Formatted at 2026-09-26 16:44:51 on dist-test-slave-n1hw"
server_key: "9bd2015b96e07997b162b17de979e025"
server_key_iv: "0937d75a70f0f08333df9ae20125b214"
server_key_version: "encryptionkey@0"
I20260926 16:44:51.585008 3065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance:
uuid: "9e58b0c44a5b4076b135b9a2284d2335"
format_stamp: "Formatted at 2026-09-26 16:44:51 on dist-test-slave-n1hw"
server_key: "9bd2015b96e07997b162b17de979e025"
server_key_iv: "0937d75a70f0f08333df9ae20125b214"
server_key_version: "encryptionkey@0"
I20260926 16:44:51.588685 3065 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 16:44:51.591163 3080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:51.592435 3065 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 16:44:51.592612 3065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-1/wal
uuid: "9e58b0c44a5b4076b135b9a2284d2335"
format_stamp: "Formatted at 2026-09-26 16:44:51 on dist-test-slave-n1hw"
server_key: "9bd2015b96e07997b162b17de979e025"
server_key_iv: "0937d75a70f0f08333df9ae20125b214"
server_key_version: "encryptionkey@0"
I20260926 16:44:51.592720 3065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:51.614225 3065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:51.617406 3065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:51.617609 3065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:51.618264 3065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:44:51.619071 3065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:44:51.619117 3065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:51.619197 3065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:44:51.619234 3065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:51.628226 3065 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.66:39565
I20260926 16:44:51.628247 3194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.66:39565 every 8 connection(s)
I20260926 16:44:51.629081 3065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
I20260926 16:44:51.631315 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 3065
I20260926 16:44:51.631407 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance
I20260926 16:44:51.631722 21553 external_mini_cluster.cc:1428] Setting key b1f82b71bcca53bd9b489b57c353ca0f
Sep 26 16:44:51 dist-test-slave-n1hw krb5kdc[2822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441091, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.66@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.12.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.67@KRBTEST.COM" created.
I20260926 16:44:51.644352 2907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:51.630969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:58487 (local address 127.21.12.126:43623)
0926 16:44:51.631132 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:51.631144 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:51.632075 (+ 931us) server_negotiation.cc:407] Connection header received
0926 16:44:51.633620 (+ 1545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:51.633626 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:51.633687 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:51.633804 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:51.635122 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:51.635649 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:51.637435 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:51.637651 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:51.637729 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:51.640543 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:51.640546 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:51.640548 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:51.640570 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:51.642096 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:51.642656 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:51.642660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:51.642662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:51.642743 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:51.643132 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:51.643139 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:51.643141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:51.643344 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:51.643458 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:51.644101 (+ 643us) server_negotiation.cc:299] Negotiation successful
0926 16:44:51.644237 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 16:44:51.645483 3197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:51.631339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.66:58487)
0926 16:44:51.631903 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:51.631959 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:51.633385 (+ 1426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:51.633969 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:51.633977 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:51.634331 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:51.634971 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:51.634991 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:51.635774 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:51.635781 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:51.637306 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:51.637320 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:51.637828 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:51.637835 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:51.638063 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:51.638701 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:51.638727 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:51.640371 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:51.642272 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:51.642277 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:51.642291 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:51.642547 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:51.642848 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:51.642851 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:51.642853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:51.643005 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:51.643451 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:51.643457 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:51.643746 (+ 289us) client_negotiation.cc:769] Sending connection context
0926 16:44:51.644026 (+ 280us) client_negotiation.cc:240] Negotiation successful
0926 16:44:51.644369 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":121,"threads_started":1}
I20260926 16:44:51.646680 3195 heartbeater.cc:347] Connected to a master server at 127.21.12.126:43623
I20260926 16:44:51.646925 3195 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:51.647395 3195 heartbeater.cc:511] Master 127.21.12.126:43623 requested a full tablet report, sending...
I20260926 16:44:51.648501 2865 ts_manager.cc:194] Registered new tserver with Master: 9e58b0c44a5b4076b135b9a2284d2335 (127.21.12.66:39565)
I20260926 16:44:51.649418 2865 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.66@KRBTEST.COM'} at 127.21.12.66:58487
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab.
Entry for principal kudu/127.21.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab.
I20260926 16:44:51.661725 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.67:0
--local_ip_for_outbound_sockets=127.21.12.67
--webserver_interface=127.21.12.67
--webserver_port=0
--tserver_master_addrs=127.21.12.126:43623
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab
--principal=kudu/127.21.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:38015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:44:51.771818 3200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:51.772064 3200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:51.772140 3200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:51.776075 3200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:51.776157 3200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:51.776244 3200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.67
I20260926 16:44:51.781417 3200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:38015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab
--principal=kudu/127.21.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.67
--webserver_port=0
--tserver_master_addrs=127.21.12.126:43623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.67
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:44:51.782508 3200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:51.783514 3200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:51.793147 3208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:51.793154 3206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:51.793205 3200 server_base.cc:1034] running on GCE node
W20260926 16:44:51.793432 3205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:51.793985 3200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:51.794659 3200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:51.795857 3200 hybrid_clock.cc:648] HybridClock initialized: now 1790441091795841 us; error 39 us; skew 500 ppm
Sep 26 16:44:51 dist-test-slave-n1hw krb5kdc[2822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441091, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:51.798745 3200 init.cc:377] Logged in from keytab as kudu/127.21.12.67@KRBTEST.COM (short username kudu)
I20260926 16:44:51.799618 3200 webserver.cc:460] Webserver started at http://127.21.12.67:41193/ using document root <none> and password file <none>
I20260926 16:44:51.800112 3200 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:51.800168 3200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:51.800323 3200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:44:51.802313 3200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-2/data/instance:
uuid: "cfd6bcea69494c108b9e46a933425537"
format_stamp: "Formatted at 2026-09-26 16:44:51 on dist-test-slave-n1hw"
server_key: "1f51aea165f163a1d2e3d21f569c8c54"
server_key_iv: "880a5223aa2074af0da259943003433b"
server_key_version: "encryptionkey@0"
I20260926 16:44:51.802829 3200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance:
uuid: "cfd6bcea69494c108b9e46a933425537"
format_stamp: "Formatted at 2026-09-26 16:44:51 on dist-test-slave-n1hw"
server_key: "1f51aea165f163a1d2e3d21f569c8c54"
server_key_iv: "880a5223aa2074af0da259943003433b"
server_key_version: "encryptionkey@0"
I20260926 16:44:51.806401 3200 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:44:51.808725 3215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:51.810014 3200 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:44:51.810129 3200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-2/wal
uuid: "cfd6bcea69494c108b9e46a933425537"
format_stamp: "Formatted at 2026-09-26 16:44:51 on dist-test-slave-n1hw"
server_key: "1f51aea165f163a1d2e3d21f569c8c54"
server_key_iv: "880a5223aa2074af0da259943003433b"
server_key_version: "encryptionkey@0"
I20260926 16:44:51.810240 3200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:51.830075 3200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:51.833269 3200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:51.833473 3200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:51.834120 3200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:44:51.834900 3200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:44:51.834944 3200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:51.834982 3200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:44:51.835043 3200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:51.843698 3200 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.67:46679
I20260926 16:44:51.843720 3329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.67:46679 every 8 connection(s)
I20260926 16:44:51.844547 3200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
I20260926 16:44:51.847754 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 3200
I20260926 16:44:51.847893 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance
I20260926 16:44:51.848227 21553 external_mini_cluster.cc:1428] Setting key 357b848b4fdb498bf8c9f8357cb6a67e
Sep 26 16:44:51 dist-test-slave-n1hw krb5kdc[2822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441091, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.67@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:44:51.859050 2907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:51.846411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:43419 (local address 127.21.12.126:43623)
0926 16:44:51.846559 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:51.846564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:51.847348 (+ 784us) server_negotiation.cc:407] Connection header received
0926 16:44:51.849148 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:51.849155 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:51.849216 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:51.849301 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:51.850820 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:51.851408 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:51.852614 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:51.852852 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:51.852952 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:51.855476 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:51.855479 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:51.855481 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:51.855506 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:51.856852 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:51.857364 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:51.857367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:51.857368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:51.857443 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:51.857914 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:51.857917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:51.857918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:51.858107 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:51.858186 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:51.858769 (+ 583us) server_negotiation.cc:299] Negotiation successful
0926 16:44:51.858883 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 16:44:51.859827 3332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:51.846725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.67:43419)
0926 16:44:51.847199 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:51.847240 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:51.848880 (+ 1640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:51.849480 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:51.849487 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:51.850017 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:51.850687 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:51.850704 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:51.851519 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:51.851521 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:51.852477 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:51.852486 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:51.852989 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:51.852991 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:51.853147 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:51.853733 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:51.853816 (+ 83us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:51.855309 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:51.856985 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:51.856990 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:51.857001 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:51.857267 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:51.857570 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:51.857573 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:51.857574 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:51.857719 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:51.858212 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:51.858218 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:51.858484 (+ 266us) client_negotiation.cc:769] Sending connection context
0926 16:44:51.858649 (+ 165us) client_negotiation.cc:240] Negotiation successful
0926 16:44:51.858902 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":135,"threads_started":1}
I20260926 16:44:51.861207 3330 heartbeater.cc:347] Connected to a master server at 127.21.12.126:43623
I20260926 16:44:51.861495 3330 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:51.862146 3330 heartbeater.cc:511] Master 127.21.12.126:43623 requested a full tablet report, sending...
I20260926 16:44:51.863276 2865 ts_manager.cc:194] Registered new tserver with Master: cfd6bcea69494c108b9e46a933425537 (127.21.12.67:46679)
I20260926 16:44:51.863839 2865 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.67@KRBTEST.COM'} at 127.21.12.67:43419
I20260926 16:44:51.874356 21553 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:44:51.883878 2907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:51.876315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37880 (local address 127.21.12.126:43623)
0926 16:44:51.876460 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:51.876463 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:51.876683 (+ 220us) server_negotiation.cc:407] Connection header received
0926 16:44:51.876785 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:51.876787 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:51.876825 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:51.876899 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:51.877655 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:51.878214 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:51.879162 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:51.879379 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:51.879529 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:51.880945 (+ 1416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:51.880948 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:51.880950 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:51.880971 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:51.882390 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:51.882772 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:51.882774 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:51.882776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:51.882849 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:51.883118 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:51.883120 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:51.883121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:51.883315 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:51.883416 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:51.883677 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 16:44:51.883765 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 16:44:51.886656 2865 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37880:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:44:51.889613 2865 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:44:51.904781 3344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:51.899224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:39565 (local address 127.0.0.1:50032)
0926 16:44:51.899918 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:51.899933 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:51.900035 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:51.900707 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:51.900710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:51.900729 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:44:51.900837 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:51.900847 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:51.901965 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:51.901968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:51.903258 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:51.903266 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:51.904297 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:51.904304 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:51.904426 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:51.904463 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 16:44:51.904541 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 16:44:51.904613 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"mutex_wait_us":116,"thread_start_us":87,"threads_started":1}
I20260926 16:44:51.904834 3343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:51.898627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.67:46679 (local address 127.0.0.1:47116)
0926 16:44:51.899102 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:51.899118 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:51.899214 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:51.900221 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:51.900224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:51.900246 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:44:51.900351 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:51.900357 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:51.901876 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:51.901881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:51.903343 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:51.903352 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:51.904478 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:51.904483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:51.904592 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:51.904611 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 16:44:51.904660 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 16:44:51.904717 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"mutex_wait_us":107,"thread_start_us":97,"threads_started":1}
I20260926 16:44:51.904956 3342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:51.898521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52732 (local address 127.21.12.65:41567)
0926 16:44:51.899296 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:51.899314 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:51.899340 (+ 26us) server_negotiation.cc:407] Connection header received
0926 16:44:51.899423 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:51.899431 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:51.899679 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:51.899848 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:44:51.900348 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:51.901558 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:51.902986 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:51.903900 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:51.904026 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:51.904179 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:51.904694 (+ 515us) server_negotiation.cc:299] Negotiation successful
0926 16:44:51.904786 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":120,"threads_started":1}
I20260926 16:44:51.904632 3341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:51.898367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:41567 (local address 127.0.0.1:52732)
0926 16:44:51.898747 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:51.898799 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:51.898915 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:51.900056 (+ 1141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:51.900065 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:51.900092 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:44:51.900238 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:51.900245 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:51.901698 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:51.901701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:51.902844 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:51.902854 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:51.904034 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:51.904038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:51.904151 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:51.904209 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 16:44:51.904311 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 16:44:51.904434 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":39,"thread_start_us":94,"threads_started":1}
I20260926 16:44:51.904830 3346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:51.899516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50032 (local address 127.21.12.66:39565)
0926 16:44:51.900305 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:51.900319 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:51.900344 (+ 25us) server_negotiation.cc:407] Connection header received
0926 16:44:51.900426 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:51.900431 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:51.900567 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:51.900714 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:44:51.900944 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:51.901836 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:51.903390 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:51.904138 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:51.904294 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:51.904434 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:51.904586 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 16:44:51.904691 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":131,"threads_started":1}
I20260926 16:44:51.905597 3345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:51.899190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47116 (local address 127.21.12.67:46679)
0926 16:44:51.899783 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:51.899795 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:51.899815 (+ 20us) server_negotiation.cc:407] Connection header received
0926 16:44:51.899887 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:51.899891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:51.900068 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:51.900493 (+ 425us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:44:51.900644 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:51.901690 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:51.903481 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:51.904341 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:51.905009 (+ 668us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:51.905252 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:51.905314 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 16:44:51.905404 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":136,"threads_started":1}
I20260926 16:44:51.907922 2994 tablet_service.cc:1467] Processing CreateTablet for tablet 992426d3146a4c098193706997a2e2a5 (DEFAULT_TABLE table=test-table [id=e9f65566d8b34f40b3ef0d1e21df5bc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:44:51.907923 3129 tablet_service.cc:1467] Processing CreateTablet for tablet 992426d3146a4c098193706997a2e2a5 (DEFAULT_TABLE table=test-table [id=e9f65566d8b34f40b3ef0d1e21df5bc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:44:51.908874 3129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 992426d3146a4c098193706997a2e2a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:44:51.908926 2994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 992426d3146a4c098193706997a2e2a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:44:51.910305 3264 tablet_service.cc:1467] Processing CreateTablet for tablet 992426d3146a4c098193706997a2e2a5 (DEFAULT_TABLE table=test-table [id=e9f65566d8b34f40b3ef0d1e21df5bc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:44:51.911289 3264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 992426d3146a4c098193706997a2e2a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:44:51.915774 3347 tablet_bootstrap.cc:492] T 992426d3146a4c098193706997a2e2a5 P 9e58b0c44a5b4076b135b9a2284d2335: Bootstrap starting.
I20260926 16:44:51.916950 3348 tablet_bootstrap.cc:492] T 992426d3146a4c098193706997a2e2a5 P e2f1b9f64c4d4917a133f18b471fab50: Bootstrap starting.
I20260926 16:44:51.918256 3347 tablet_bootstrap.cc:654] T 992426d3146a4c098193706997a2e2a5 P 9e58b0c44a5b4076b135b9a2284d2335: Neither blocks nor log segments found. Creating new log.
I20260926 16:44:51.919032 3347 log.cc:824] T 992426d3146a4c098193706997a2e2a5 P 9e58b0c44a5b4076b135b9a2284d2335: Log is configured to *not* fsync() on all Append() calls
I20260926 16:44:51.919248 3348 tablet_bootstrap.cc:654] T 992426d3146a4c098193706997a2e2a5 P e2f1b9f64c4d4917a133f18b471fab50: Neither blocks nor log segments found. Creating new log.
I20260926 16:44:51.919297 3349 tablet_bootstrap.cc:492] T 992426d3146a4c098193706997a2e2a5 P cfd6bcea69494c108b9e46a933425537: Bootstrap starting.
I20260926 16:44:51.920679 3348 log.cc:824] T 992426d3146a4c098193706997a2e2a5 P e2f1b9f64c4d4917a133f18b471fab50: Log is configured to *not* fsync() on all Append() calls
I20260926 16:44:51.921531 3349 tablet_bootstrap.cc:654] T 992426d3146a4c098193706997a2e2a5 P cfd6bcea69494c108b9e46a933425537: Neither blocks nor log segments found. Creating new log.
I20260926 16:44:51.922824 3349 log.cc:824] T 992426d3146a4c098193706997a2e2a5 P cfd6bcea69494c108b9e46a933425537: Log is configured to *not* fsync() on all Append() calls
I20260926 16:44:51.922989 3348 tablet_bootstrap.cc:492] T 992426d3146a4c098193706997a2e2a5 P e2f1b9f64c4d4917a133f18b471fab50: No bootstrap required, opened a new log
I20260926 16:44:51.923121 3347 tablet_bootstrap.cc:492] T 992426d3146a4c098193706997a2e2a5 P 9e58b0c44a5b4076b135b9a2284d2335: No bootstrap required, opened a new log
I20260926 16:44:51.923194 3348 ts_tablet_manager.cc:1397] T 992426d3146a4c098193706997a2e2a5 P e2f1b9f64c4d4917a133f18b471fab50: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 16:44:51.923367 3347 ts_tablet_manager.cc:1397] T 992426d3146a4c098193706997a2e2a5 P 9e58b0c44a5b4076b135b9a2284d2335: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260926 16:44:51.924734 3349 tablet_bootstrap.cc:492] T 992426d3146a4c098193706997a2e2a5 P cfd6bcea69494c108b9e46a933425537: No bootstrap required, opened a new log
I20260926 16:44:51.924908 3349 ts_tablet_manager.cc:1397] T 992426d3146a4c098193706997a2e2a5 P cfd6bcea69494c108b9e46a933425537: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 16:44:51.926376 3347 raft_consensus.cc:348] T 992426d3146a4c098193706997a2e2a5 P 9e58b0c44a5b4076b135b9a2284d2335 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f1b9f64c4d4917a133f18b471fab50" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41567 } } peers { permanent_uuid: "cfd6bcea69494c108b9e46a933425537" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 46679 } } peers { permanent_uuid: "9e58b0c44a5b4076b135b9a2284d2335" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 39565 } }
I20260926 16:44:51.926614 3347 raft_consensus.cc:374] T 992426d3146a4c098193706997a2e2a5 P 9e58b0c44a5b4076b135b9a2284d2335 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:44:51.926683 3347 raft_consensus.cc:729] T 992426d3146a4c098193706997a2e2a5 P 9e58b0c44a5b4076b135b9a2284d2335 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e58b0c44a5b4076b135b9a2284d2335, State: Initialized, Role: FOLLOWER
I20260926 16:44:51.927145 3347 consensus_queue.cc:260] T 992426d3146a4c098193706997a2e2a5 P 9e58b0c44a5b4076b135b9a2284d2335 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f1b9f64c4d4917a133f18b471fab50" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41567 } } peers { permanent_uuid: "cfd6bcea69494c108b9e46a933425537" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 46679 } } peers { permanent_uuid: "9e58b0c44a5b4076b135b9a2284d2335" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 39565 } }
I20260926 16:44:51.927378 3348 raft_consensus.cc:348] T 992426d3146a4c098193706997a2e2a5 P e2f1b9f64c4d4917a133f18b471fab50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f1b9f64c4d4917a133f18b471fab50" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41567 } } peers { permanent_uuid: "cfd6bcea69494c108b9e46a933425537" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 46679 } } peers { permanent_uuid: "9e58b0c44a5b4076b135b9a2284d2335" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 39565 } }
I20260926 16:44:51.927716 3348 raft_consensus.cc:374] T 992426d3146a4c098193706997a2e2a5 P e2f1b9f64c4d4917a133f18b471fab50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:44:51.927829 3348 raft_consensus.cc:729] T 992426d3146a4c098193706997a2e2a5 P e2f1b9f64c4d4917a133f18b471fab50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2f1b9f64c4d4917a133f18b471fab50, State: Initialized, Role: FOLLOWER
I20260926 16:44:51.928095 3195 heartbeater.cc:503] Master 127.21.12.126:43623 was elected leader, sending a full tablet report...
I20260926 16:44:51.928241 3347 ts_tablet_manager.cc:1428] T 992426d3146a4c098193706997a2e2a5 P 9e58b0c44a5b4076b135b9a2284d2335: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20260926 16:44:51.928303 3348 consensus_queue.cc:260] T 992426d3146a4c098193706997a2e2a5 P e2f1b9f64c4d4917a133f18b471fab50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f1b9f64c4d4917a133f18b471fab50" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41567 } } peers { permanent_uuid: "cfd6bcea69494c108b9e46a933425537" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 46679 } } peers { permanent_uuid: "9e58b0c44a5b4076b135b9a2284d2335" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 39565 } }
I20260926 16:44:51.928882 3060 heartbeater.cc:503] Master 127.21.12.126:43623 was elected leader, sending a full tablet report...
I20260926 16:44:51.929270 3348 ts_tablet_manager.cc:1428] T 992426d3146a4c098193706997a2e2a5 P e2f1b9f64c4d4917a133f18b471fab50: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 16:44:51.930371 3349 raft_consensus.cc:348] T 992426d3146a4c098193706997a2e2a5 P cfd6bcea69494c108b9e46a933425537 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f1b9f64c4d4917a133f18b471fab50" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41567 } } peers { permanent_uuid: "cfd6bcea69494c108b9e46a933425537" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 46679 } } peers { permanent_uuid: "9e58b0c44a5b4076b135b9a2284d2335" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 39565 } }
I20260926 16:44:51.930655 3349 raft_consensus.cc:374] T 992426d3146a4c098193706997a2e2a5 P cfd6bcea69494c108b9e46a933425537 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:44:51.930755 3349 raft_consensus.cc:729] T 992426d3146a4c098193706997a2e2a5 P cfd6bcea69494c108b9e46a933425537 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfd6bcea69494c108b9e46a933425537, State: Initialized, Role: FOLLOWER
I20260926 16:44:51.931191 3349 consensus_queue.cc:260] T 992426d3146a4c098193706997a2e2a5 P cfd6bcea69494c108b9e46a933425537 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f1b9f64c4d4917a133f18b471fab50" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41567 } } peers { permanent_uuid: "cfd6bcea69494c108b9e46a933425537" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 46679 } } peers { permanent_uuid: "9e58b0c44a5b4076b135b9a2284d2335" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 39565 } }
I20260926 16:44:51.932128 3330 heartbeater.cc:503] Master 127.21.12.126:43623 was elected leader, sending a full tablet report...
I20260926 16:44:51.933085 3349 ts_tablet_manager.cc:1428] T 992426d3146a4c098193706997a2e2a5 P cfd6bcea69494c108b9e46a933425537: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
W20260926 16:44:52.095468 3331 tablet.cc:2369] T 992426d3146a4c098193706997a2e2a5 P cfd6bcea69494c108b9e46a933425537: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:44:52.130232 3196 tablet.cc:2369] T 992426d3146a4c098193706997a2e2a5 P 9e58b0c44a5b4076b135b9a2284d2335: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:44:52.152949 3355 raft_consensus.cc:482] T 992426d3146a4c098193706997a2e2a5 P cfd6bcea69494c108b9e46a933425537 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:44:52.153169 3355 raft_consensus.cc:504] T 992426d3146a4c098193706997a2e2a5 P cfd6bcea69494c108b9e46a933425537 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f1b9f64c4d4917a133f18b471fab50" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41567 } } peers { permanent_uuid: "cfd6bcea69494c108b9e46a933425537" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 46679 } } peers { permanent_uuid: "9e58b0c44a5b4076b135b9a2284d2335" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 39565 } }
I20260926 16:44:52.154330 3355 leader_election.cc:288] T 992426d3146a4c098193706997a2e2a5 P cfd6bcea69494c108b9e46a933425537 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2f1b9f64c4d4917a133f18b471fab50 (127.21.12.65:41567), 9e58b0c44a5b4076b135b9a2284d2335 (127.21.12.66:39565)
I20260926 16:44:52.158058 3342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:52.154808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:48649 (local address 127.21.12.65:41567)
0926 16:44:52.154957 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:52.154963 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:52.154979 (+ 16us) server_negotiation.cc:407] Connection header received
0926 16:44:52.155139 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:52.155144 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:52.155198 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:52.155279 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:44:52.155528 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:52.156059 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:52.157100 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:52.157484 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:52.157610 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:52.157661 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:52.157857 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 16:44:52.157927 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 16:44:52.158064 3332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:52.154717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:41567 (local address 127.21.12.67:48649)
0926 16:44:52.154871 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:52.154889 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:52.155002 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:52.155288 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:52.155291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:52.155308 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:44:52.155422 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:52.155428 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:52.156167 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:52.156171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:52.156986 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:52.156995 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:52.157599 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:52.157602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:52.157713 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:52.157730 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 16:44:52.157844 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 16:44:52.157917 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 16:44:52.158458 3346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:52.155132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:39643 (local address 127.21.12.66:39565)
0926 16:44:52.155298 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:52.155303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:52.155440 (+ 137us) server_negotiation.cc:407] Connection header received
0926 16:44:52.155563 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:52.155566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:52.155626 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:52.155712 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:44:52.155982 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:52.156460 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:52.157488 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:52.157907 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:52.158036 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:52.158083 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:52.158291 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 16:44:52.158356 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260926 16:44:52.158471 3356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:52.155005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:39565 (local address 127.21.12.67:39643)
0926 16:44:52.155343 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:52.155358 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:52.155477 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:52.155724 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:52.155727 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:52.155741 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:44:52.155869 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:52.155875 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:52.156579 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:52.156582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:52.157366 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:52.157374 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:52.158028 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:52.158032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:52.158139 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:52.158166 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 16:44:52.158274 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 16:44:52.158340 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":121,"threads_started":1}
I20260926 16:44:52.158677 3014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992426d3146a4c098193706997a2e2a5" candidate_uuid: "cfd6bcea69494c108b9e46a933425537" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f1b9f64c4d4917a133f18b471fab50" is_pre_election: true
I20260926 16:44:52.159032 3014 raft_consensus.cc:2454] T 992426d3146a4c098193706997a2e2a5 P e2f1b9f64c4d4917a133f18b471fab50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfd6bcea69494c108b9e46a933425537 in term 0.
I20260926 16:44:52.159133 3149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992426d3146a4c098193706997a2e2a5" candidate_uuid: "cfd6bcea69494c108b9e46a933425537" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e58b0c44a5b4076b135b9a2284d2335" is_pre_election: true
I20260926 16:44:52.159404 3149 raft_consensus.cc:2454] T 992426d3146a4c098193706997a2e2a5 P 9e58b0c44a5b4076b135b9a2284d2335 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfd6bcea69494c108b9e46a933425537 in term 0.
I20260926 16:44:52.159495 3220 leader_election.cc:302] T 992426d3146a4c098193706997a2e2a5 P cfd6bcea69494c108b9e46a933425537 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cfd6bcea69494c108b9e46a933425537, e2f1b9f64c4d4917a133f18b471fab50; no voters:
I20260926 16:44:52.159804 3355 raft_consensus.cc:2781] T 992426d3146a4c098193706997a2e2a5 P cfd6bcea69494c108b9e46a933425537 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:44:52.159894 3355 raft_consensus.cc:482] T 992426d3146a4c098193706997a2e2a5 P cfd6bcea69494c108b9e46a933425537 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:44:52.159940 3355 raft_consensus.cc:3037] T 992426d3146a4c098193706997a2e2a5 P cfd6bcea69494c108b9e46a933425537 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:44:52.160939 3355 raft_consensus.cc:504] T 992426d3146a4c098193706997a2e2a5 P cfd6bcea69494c108b9e46a933425537 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f1b9f64c4d4917a133f18b471fab50" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41567 } } peers { permanent_uuid: "cfd6bcea69494c108b9e46a933425537" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 46679 } } peers { permanent_uuid: "9e58b0c44a5b4076b135b9a2284d2335" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 39565 } }
I20260926 16:44:52.161268 3355 leader_election.cc:288] T 992426d3146a4c098193706997a2e2a5 P cfd6bcea69494c108b9e46a933425537 [CANDIDATE]: Term 1 election: Requested vote from peers e2f1b9f64c4d4917a133f18b471fab50 (127.21.12.65:41567), 9e58b0c44a5b4076b135b9a2284d2335 (127.21.12.66:39565)
I20260926 16:44:52.161652 3014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992426d3146a4c098193706997a2e2a5" candidate_uuid: "cfd6bcea69494c108b9e46a933425537" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f1b9f64c4d4917a133f18b471fab50"
I20260926 16:44:52.161665 3149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992426d3146a4c098193706997a2e2a5" candidate_uuid: "cfd6bcea69494c108b9e46a933425537" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e58b0c44a5b4076b135b9a2284d2335"
I20260926 16:44:52.161800 3014 raft_consensus.cc:3037] T 992426d3146a4c098193706997a2e2a5 P e2f1b9f64c4d4917a133f18b471fab50 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:44:52.161800 3149 raft_consensus.cc:3037] T 992426d3146a4c098193706997a2e2a5 P 9e58b0c44a5b4076b135b9a2284d2335 [term 0 FOLLOWER]: Advancing to term 1
W20260926 16:44:52.162089 3061 tablet.cc:2369] T 992426d3146a4c098193706997a2e2a5 P e2f1b9f64c4d4917a133f18b471fab50: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:44:52.163025 3149 raft_consensus.cc:2454] T 992426d3146a4c098193706997a2e2a5 P 9e58b0c44a5b4076b135b9a2284d2335 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfd6bcea69494c108b9e46a933425537 in term 1.
I20260926 16:44:52.163025 3014 raft_consensus.cc:2454] T 992426d3146a4c098193706997a2e2a5 P e2f1b9f64c4d4917a133f18b471fab50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfd6bcea69494c108b9e46a933425537 in term 1.
I20260926 16:44:52.163321 3220 leader_election.cc:302] T 992426d3146a4c098193706997a2e2a5 P cfd6bcea69494c108b9e46a933425537 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cfd6bcea69494c108b9e46a933425537, e2f1b9f64c4d4917a133f18b471fab50; no voters:
I20260926 16:44:52.163476 3355 raft_consensus.cc:2781] T 992426d3146a4c098193706997a2e2a5 P cfd6bcea69494c108b9e46a933425537 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:44:52.163733 3355 raft_consensus.cc:686] T 992426d3146a4c098193706997a2e2a5 P cfd6bcea69494c108b9e46a933425537 [term 1 LEADER]: Becoming Leader. State: Replica: cfd6bcea69494c108b9e46a933425537, State: Running, Role: LEADER
I20260926 16:44:52.164049 3355 consensus_queue.cc:237] T 992426d3146a4c098193706997a2e2a5 P cfd6bcea69494c108b9e46a933425537 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f1b9f64c4d4917a133f18b471fab50" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41567 } } peers { permanent_uuid: "cfd6bcea69494c108b9e46a933425537" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 46679 } } peers { permanent_uuid: "9e58b0c44a5b4076b135b9a2284d2335" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 39565 } }
I20260926 16:44:52.166989 2865 catalog_manager.cc:5495] T 992426d3146a4c098193706997a2e2a5 P cfd6bcea69494c108b9e46a933425537 reported cstate change: term changed from 0 to 1, leader changed from <none> to cfd6bcea69494c108b9e46a933425537 (127.21.12.67). New cstate: current_term: 1 leader_uuid: "cfd6bcea69494c108b9e46a933425537" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f1b9f64c4d4917a133f18b471fab50" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfd6bcea69494c108b9e46a933425537" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 46679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e58b0c44a5b4076b135b9a2284d2335" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 39565 } health_report { overall_health: UNKNOWN } } }
I20260926 16:44:52.194568 3345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:52.190974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47120 (local address 127.21.12.67:46679)
0926 16:44:52.191135 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:52.191141 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:52.191157 (+ 16us) server_negotiation.cc:407] Connection header received
0926 16:44:52.191312 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:52.191314 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:52.191365 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:52.191455 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:44:52.191669 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:52.192170 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:52.193015 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:52.193227 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:52.193307 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:52.193646 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:44:52.194085 (+ 439us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:44:52.194177 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:52.194283 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 16:44:52.194329 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 16:44:52.200430 3014 raft_consensus.cc:1261] T 992426d3146a4c098193706997a2e2a5 P e2f1b9f64c4d4917a133f18b471fab50 [term 1 FOLLOWER]: Refusing update from remote peer cfd6bcea69494c108b9e46a933425537: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:44:52.200430 3149 raft_consensus.cc:1261] T 992426d3146a4c098193706997a2e2a5 P 9e58b0c44a5b4076b135b9a2284d2335 [term 1 FOLLOWER]: Refusing update from remote peer cfd6bcea69494c108b9e46a933425537: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:44:52.201225 3357 consensus_queue.cc:1035] T 992426d3146a4c098193706997a2e2a5 P cfd6bcea69494c108b9e46a933425537 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e58b0c44a5b4076b135b9a2284d2335" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 39565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:44:52.201479 3355 consensus_queue.cc:1035] T 992426d3146a4c098193706997a2e2a5 P cfd6bcea69494c108b9e46a933425537 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2f1b9f64c4d4917a133f18b471fab50" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:44:52.209797 3362 mvcc.cc:205] Tried to move back new op lower bound from 7333646713645305856 to 7333646713506353152. Current Snapshot: MvccSnapshot[applied={T|T < 7333646713645305856}]
I20260926 16:44:52.217507 2868 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:44:52.217659 2868 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:44:52.220005 2868 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17: Sending DeleteTablet for 3 replicas of tablet 992426d3146a4c098193706997a2e2a5
I20260926 16:44:52.220770 3264 tablet_service.cc:1514] Processing DeleteTablet for tablet 992426d3146a4c098193706997a2e2a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:44:52 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:47116
I20260926 16:44:52.220779 2994 tablet_service.cc:1514] Processing DeleteTablet for tablet 992426d3146a4c098193706997a2e2a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:44:52 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:52732
I20260926 16:44:52.220825 3129 tablet_service.cc:1514] Processing DeleteTablet for tablet 992426d3146a4c098193706997a2e2a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:44:52 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:50032
I20260926 16:44:52.221244 3371 tablet_replica.cc:330] T 992426d3146a4c098193706997a2e2a5 P e2f1b9f64c4d4917a133f18b471fab50: stopping tablet replica
I20260926 16:44:52.221266 3370 tablet_replica.cc:330] T 992426d3146a4c098193706997a2e2a5 P 9e58b0c44a5b4076b135b9a2284d2335: stopping tablet replica
I20260926 16:44:52.221424 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 2930
I20260926 16:44:52.221545 3370 raft_consensus.cc:2229] T 992426d3146a4c098193706997a2e2a5 P 9e58b0c44a5b4076b135b9a2284d2335 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:44:52.221859 3370 raft_consensus.cc:2258] T 992426d3146a4c098193706997a2e2a5 P 9e58b0c44a5b4076b135b9a2284d2335 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:44:52.223344 3370 ts_tablet_manager.cc:1905] T 992426d3146a4c098193706997a2e2a5 P 9e58b0c44a5b4076b135b9a2284d2335: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:44:52.225988 3372 tablet_replica.cc:330] T 992426d3146a4c098193706997a2e2a5 P cfd6bcea69494c108b9e46a933425537: stopping tablet replica
I20260926 16:44:52.226297 3372 raft_consensus.cc:2229] T 992426d3146a4c098193706997a2e2a5 P cfd6bcea69494c108b9e46a933425537 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:44:52.226693 3372 raft_consensus.cc:2258] T 992426d3146a4c098193706997a2e2a5 P cfd6bcea69494c108b9e46a933425537 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:44:52.226718 3370 ts_tablet_manager.cc:1918] T 992426d3146a4c098193706997a2e2a5 P 9e58b0c44a5b4076b135b9a2284d2335: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:44:52.226861 3370 log.cc:1196] T 992426d3146a4c098193706997a2e2a5 P 9e58b0c44a5b4076b135b9a2284d2335: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-1/wal/wals/992426d3146a4c098193706997a2e2a5
I20260926 16:44:52.227345 3370 ts_tablet_manager.cc:1939] T 992426d3146a4c098193706997a2e2a5 P 9e58b0c44a5b4076b135b9a2284d2335: Deleting consensus metadata
I20260926 16:44:52.228135 3372 ts_tablet_manager.cc:1905] T 992426d3146a4c098193706997a2e2a5 P cfd6bcea69494c108b9e46a933425537: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:44:52.228622 2854 catalog_manager.cc:4841] TS 9e58b0c44a5b4076b135b9a2284d2335 (127.21.12.66:39565): tablet 992426d3146a4c098193706997a2e2a5 (table test-table [id=e9f65566d8b34f40b3ef0d1e21df5bc9]) successfully deleted
W20260926 16:44:52.229728 2854 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.12.65:41567 (error 108)
I20260926 16:44:52.229990 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 3065
I20260926 16:44:52.231849 3341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:52.231581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:41567 (local address 127.0.0.1:52746)
0926 16:44:52.231697 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:52.231769 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.65:41567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 16:44:52.232079 3372 ts_tablet_manager.cc:1918] T 992426d3146a4c098193706997a2e2a5 P cfd6bcea69494c108b9e46a933425537: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:44:52.232200 3372 log.cc:1196] T 992426d3146a4c098193706997a2e2a5 P cfd6bcea69494c108b9e46a933425537: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-2/wal/wals/992426d3146a4c098193706997a2e2a5
I20260926 16:44:52.232491 3372 ts_tablet_manager.cc:1939] T 992426d3146a4c098193706997a2e2a5 P cfd6bcea69494c108b9e46a933425537: Deleting consensus metadata
I20260926 16:44:52.233425 2851 catalog_manager.cc:4841] TS cfd6bcea69494c108b9e46a933425537 (127.21.12.67:46679): tablet 992426d3146a4c098193706997a2e2a5 (table test-table [id=e9f65566d8b34f40b3ef0d1e21df5bc9]) successfully deleted
W20260926 16:44:52.233635 2854 catalog_manager.cc:4584] TS e2f1b9f64c4d4917a133f18b471fab50 (127.21.12.65:41567): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 992426d3146a4c098193706997a2e2a5: Network error: Client connection negotiation failed: client connection to 127.21.12.65:41567: connect: Connection refused (error 111)
I20260926 16:44:52.237341 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 3200
I20260926 16:44:52.243176 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 2835
I20260926 16:44:52.249976 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:43623
--webserver_interface=127.21.12.126
--webserver_port=42949
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:38015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:43623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:44:52.362174 3374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:52.362454 3374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:52.362548 3374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:52.366514 3374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:44:52.366586 3374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:52.366619 3374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:52.366675 3374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:44:52.366714 3374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:44:52.371904 3374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:38015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:43623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:43623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.126
--webserver_port=42949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:44:52.372892 3374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:52.373999 3374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:52.387012 3380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:52.387048 3383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:52.387042 3381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:52.387113 3374 server_base.cc:1034] running on GCE node
I20260926 16:44:52.387787 3374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:52.388931 3374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:52.390117 3374 hybrid_clock.cc:648] HybridClock initialized: now 1790441092390100 us; error 29 us; skew 500 ppm
Sep 26 16:44:52 dist-test-slave-n1hw krb5kdc[2822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441092, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:52.392892 3374 init.cc:377] Logged in from keytab as kudu/127.21.12.126@KRBTEST.COM (short username kudu)
I20260926 16:44:52.393682 3374 webserver.cc:460] Webserver started at http://127.21.12.126:42949/ using document root <none> and password file <none>
I20260926 16:44:52.394246 3374 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:52.394316 3374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:52.398141 3374 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260926 16:44:52.400228 3391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:52.401538 3374 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:44:52.401643 3374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "e3ea10b58512425eb10bf0d7a55d3e17"
format_stamp: "Formatted at 2026-09-26 16:44:51 on dist-test-slave-n1hw"
server_key: "60ea5f920d55bd0c05a5067892acc098"
server_key_iv: "4b0e4304b593a2a1d6bf2f650ffcc8d2"
server_key_version: "encryptionkey@0"
I20260926 16:44:52.402279 3374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:52.420557 3374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:52.423776 3374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:52.423940 3374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:52.431305 3374 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.126:43623
I20260926 16:44:52.431599 3444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.126:43623 every 8 connection(s)
I20260926 16:44:52.432430 3374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:44:52.436185 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 3374
I20260926 16:44:52.436610 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.65:41567
--local_ip_for_outbound_sockets=127.21.12.65
--tserver_master_addrs=127.21.12.126:43623
--webserver_port=44711
--webserver_interface=127.21.12.65
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab
--principal=kudu/127.21.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:38015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:44:52.436755 3445 sys_catalog.cc:263] Verifying existing consensus state
I20260926 16:44:52.438390 3445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17: Bootstrap starting.
I20260926 16:44:52.463243 3445 log.cc:824] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17: Log is configured to *not* fsync() on all Append() calls
I20260926 16:44:52.474808 3445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 16:44:52.475293 3445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17: Bootstrap complete.
I20260926 16:44:52.481030 3445 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ea10b58512425eb10bf0d7a55d3e17" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 43623 } }
I20260926 16:44:52.481590 3445 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3ea10b58512425eb10bf0d7a55d3e17, State: Initialized, Role: FOLLOWER
I20260926 16:44:52.482221 3445 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ea10b58512425eb10bf0d7a55d3e17" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 43623 } }
I20260926 16:44:52.482396 3445 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:44:52.482476 3445 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:44:52.482591 3445 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [term 1 FOLLOWER]: Advancing to term 2
I20260926 16:44:52.485229 3445 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ea10b58512425eb10bf0d7a55d3e17" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 43623 } }
I20260926 16:44:52.485620 3445 leader_election.cc:302] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3ea10b58512425eb10bf0d7a55d3e17; no voters:
I20260926 16:44:52.485989 3445 leader_election.cc:288] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 16:44:52.486174 3449 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 16:44:52.486478 3449 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [term 2 LEADER]: Becoming Leader. State: Replica: e3ea10b58512425eb10bf0d7a55d3e17, State: Running, Role: LEADER
I20260926 16:44:52.486796 3449 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ea10b58512425eb10bf0d7a55d3e17" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 43623 } }
I20260926 16:44:52.487321 3445 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:44:52.488355 3450 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e3ea10b58512425eb10bf0d7a55d3e17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ea10b58512425eb10bf0d7a55d3e17" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 43623 } } }
I20260926 16:44:52.488355 3451 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3ea10b58512425eb10bf0d7a55d3e17. Latest consensus state: current_term: 2 leader_uuid: "e3ea10b58512425eb10bf0d7a55d3e17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ea10b58512425eb10bf0d7a55d3e17" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 43623 } } }
I20260926 16:44:52.488533 3450 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [sys.catalog]: This master's current role is: LEADER
I20260926 16:44:52.488533 3451 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [sys.catalog]: This master's current role is: LEADER
I20260926 16:44:52.489099 3458 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:44:52.493854 3458 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:44:52.494758 3458 catalog_manager.cc:1259] Loaded cluster ID: 6acfdeb002a9424981a3f8091bcff7cc
I20260926 16:44:52.494825 3458 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:44:52.496384 3458 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 16:44:52.501243 3458 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 16:44:52.506356 3458 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:44:52.507045 3458 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17: Loaded TSK: 0
I20260926 16:44:52.508172 3458 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 16:44:52.566817 3447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:52.567094 3447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:52.567168 3447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:52.571158 3447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:52.571234 3447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:52.571318 3447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.65
I20260926 16:44:52.576584 3447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:38015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab
--principal=kudu/127.21.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.65:41567
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.65
--webserver_port=44711
--tserver_master_addrs=127.21.12.126:43623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.65
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:44:52.577553 3447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:52.578567 3447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:52.589026 3474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:52.589120 3476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:52.589031 3473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:52.589617 3447 server_base.cc:1034] running on GCE node
I20260926 16:44:52.590020 3447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:52.590644 3447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:52.591845 3447 hybrid_clock.cc:648] HybridClock initialized: now 1790441092591822 us; error 39 us; skew 500 ppm
Sep 26 16:44:52 dist-test-slave-n1hw krb5kdc[2822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441092, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:52.599129 3447 init.cc:377] Logged in from keytab as kudu/127.21.12.65@KRBTEST.COM (short username kudu)
I20260926 16:44:52.599996 3447 webserver.cc:460] Webserver started at http://127.21.12.65:44711/ using document root <none> and password file <none>
I20260926 16:44:52.600535 3447 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:52.600606 3447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:52.604446 3447 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:44:52.606849 3483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:52.608148 3447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:44:52.608291 3447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-0/wal
uuid: "e2f1b9f64c4d4917a133f18b471fab50"
format_stamp: "Formatted at 2026-09-26 16:44:51 on dist-test-slave-n1hw"
server_key: "afaceace097be71f67d76240e609df8c"
server_key_iv: "5c7e68d2ff7d108238f7e847e8bc4e2a"
server_key_version: "encryptionkey@0"
I20260926 16:44:52.608817 3447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:52.643082 3447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:52.646239 3447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:52.646435 3447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:52.647049 3447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:44:52.648164 3490 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 16:44:52.652220 3447 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 16:44:52.652297 3447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260926 16:44:52.652391 3447 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 16:44:52.654112 3447 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 16:44:52.654183 3447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260926 16:44:52.654466 3490 tablet_bootstrap.cc:492] T 992426d3146a4c098193706997a2e2a5 P e2f1b9f64c4d4917a133f18b471fab50: Bootstrap starting.
I20260926 16:44:52.663970 3447 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.65:41567
I20260926 16:44:52.664036 3598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.65:41567 every 8 connection(s)
I20260926 16:44:52.664865 3447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
I20260926 16:44:52.673362 3490 log.cc:824] T 992426d3146a4c098193706997a2e2a5 P e2f1b9f64c4d4917a133f18b471fab50: Log is configured to *not* fsync() on all Append() calls
I20260926 16:44:52.674036 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 3447
I20260926 16:44:52.674541 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.66:39565
--local_ip_for_outbound_sockets=127.21.12.66
--tserver_master_addrs=127.21.12.126:43623
--webserver_port=41773
--webserver_interface=127.21.12.66
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab
--principal=kudu/127.21.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:38015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 16:44:52 dist-test-slave-n1hw krb5kdc[2822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441092, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:44:52.682879 3490 tablet_bootstrap.cc:492] T 992426d3146a4c098193706997a2e2a5 P e2f1b9f64c4d4917a133f18b471fab50: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 16:44:52.683348 3490 tablet_bootstrap.cc:492] T 992426d3146a4c098193706997a2e2a5 P e2f1b9f64c4d4917a133f18b471fab50: Bootstrap complete.
I20260926 16:44:52.683367 3601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:52.666753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.65:41919)
0926 16:44:52.667391 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:52.667439 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:52.668203 (+ 764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:52.669492 (+ 1289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:52.669501 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:52.669890 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:52.670342 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:52.670354 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:52.672376 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:52.672379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:52.673218 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:52.673227 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:52.674279 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:52.674282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:52.674416 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:52.675188 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:52.675208 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:52.678038 (+ 2830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:52.680243 (+ 2205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:52.680249 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:52.680261 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:52.680518 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:52.680866 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:52.680870 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:52.680872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:52.681007 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:52.681561 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:52.681567 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:52.681882 (+ 315us) client_negotiation.cc:769] Sending connection context
0926 16:44:52.682106 (+ 224us) client_negotiation.cc:240] Negotiation successful
0926 16:44:52.682437 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":162,"threads_started":1}
I20260926 16:44:52.683354 3602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:52.666867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:41919 (local address 127.21.12.126:43623)
0926 16:44:52.667294 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:52.667313 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:52.667559 (+ 246us) server_negotiation.cc:407] Connection header received
0926 16:44:52.668662 (+ 1103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:52.668678 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:52.669067 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:52.669461 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:52.670513 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:52.672225 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:52.673341 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:52.674106 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:52.674266 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:52.678238 (+ 3972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:52.678244 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:52.678256 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:52.678302 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:52.680124 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:52.680642 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:52.680648 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:52.680652 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:52.680762 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:52.681120 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:52.681123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:52.681125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:52.681448 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:52.681597 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:52.682207 (+ 610us) server_negotiation.cc:299] Negotiation successful
0926 16:44:52.682450 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":123,"threads_started":1}
I20260926 16:44:52.684880 3490 ts_tablet_manager.cc:1397] T 992426d3146a4c098193706997a2e2a5 P e2f1b9f64c4d4917a133f18b471fab50: Time spent bootstrapping tablet: real 0.031s user 0.008s sys 0.019s
I20260926 16:44:52.685917 3599 heartbeater.cc:347] Connected to a master server at 127.21.12.126:43623
I20260926 16:44:52.686154 3599 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:52.686625 3599 heartbeater.cc:511] Master 127.21.12.126:43623 requested a full tablet report, sending...
I20260926 16:44:52.688453 3490 raft_consensus.cc:348] T 992426d3146a4c098193706997a2e2a5 P e2f1b9f64c4d4917a133f18b471fab50 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f1b9f64c4d4917a133f18b471fab50" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41567 } } peers { permanent_uuid: "cfd6bcea69494c108b9e46a933425537" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 46679 } } peers { permanent_uuid: "9e58b0c44a5b4076b135b9a2284d2335" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 39565 } }
I20260926 16:44:52.688975 3490 raft_consensus.cc:729] T 992426d3146a4c098193706997a2e2a5 P e2f1b9f64c4d4917a133f18b471fab50 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2f1b9f64c4d4917a133f18b471fab50, State: Initialized, Role: FOLLOWER
I20260926 16:44:52.689085 3408 ts_manager.cc:194] Registered new tserver with Master: e2f1b9f64c4d4917a133f18b471fab50 (127.21.12.65:41567)
I20260926 16:44:52.689430 3490 consensus_queue.cc:260] T 992426d3146a4c098193706997a2e2a5 P e2f1b9f64c4d4917a133f18b471fab50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f1b9f64c4d4917a133f18b471fab50" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41567 } } peers { permanent_uuid: "cfd6bcea69494c108b9e46a933425537" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 46679 } } peers { permanent_uuid: "9e58b0c44a5b4076b135b9a2284d2335" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 39565 } }
I20260926 16:44:52.690443 3490 ts_tablet_manager.cc:1428] T 992426d3146a4c098193706997a2e2a5 P e2f1b9f64c4d4917a133f18b471fab50: Time spent starting tablet: real 0.005s user 0.007s sys 0.001s
I20260926 16:44:52.693187 3408 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.65@KRBTEST.COM'} at 127.21.12.65:41919
I20260926 16:44:52.695749 3599 heartbeater.cc:503] Master 127.21.12.126:43623 was elected leader, sending a full tablet report...
Sep 26 16:44:52 dist-test-slave-n1hw krb5kdc[2822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441092, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for kudu/127.21.12.65@KRBTEST.COM
I20260926 16:44:52.703521 3606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:52.692253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52762 (local address 127.21.12.65:41567)
0926 16:44:52.692571 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:52.692582 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:52.692996 (+ 414us) server_negotiation.cc:407] Connection header received
0926 16:44:52.693173 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:52.693179 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:52.693374 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:52.693508 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:52.694315 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:52.695373 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:52.696657 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:52.697160 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:52.697260 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:52.699946 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:52.699954 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:52.699963 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:52.699985 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:52.701589 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:52.702164 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:52.702169 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:52.702170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:52.702251 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:52.702571 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:52.702573 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:52.702574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:52.702811 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:52.702909 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:52.703264 (+ 355us) server_negotiation.cc:299] Negotiation successful
0926 16:44:52.703394 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":110,"threads_started":1}
I20260926 16:44:52.703526 3607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:52.692133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:41567 (local address 127.0.0.1:52762)
0926 16:44:52.692851 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:52.692897 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:52.693034 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:52.693545 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:52.693551 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:52.694019 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:52.694181 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:52.694193 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:52.695499 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:52.695503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:52.696520 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:52.696532 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:52.697272 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:52.697276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:52.697390 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:52.698078 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:52.698092 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:52.699808 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:52.701700 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:52.701709 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:52.701714 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:52.702025 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:52.702379 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:52.702382 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:52.702384 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:52.702478 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:52.702919 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:52.702925 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:52.703046 (+ 121us) client_negotiation.cc:769] Sending connection context
0926 16:44:52.703180 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 16:44:52.703359 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":231,"threads_started":1}
I20260926 16:44:52.704854 3531 tablet_service.cc:1514] Processing DeleteTablet for tablet 992426d3146a4c098193706997a2e2a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:44:52 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:52762
I20260926 16:44:52.705856 3608 tablet_replica.cc:330] stopping tablet replica
I20260926 16:44:52.706221 3608 raft_consensus.cc:2229] T 992426d3146a4c098193706997a2e2a5 P e2f1b9f64c4d4917a133f18b471fab50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:44:52.706514 3608 raft_consensus.cc:2258] T 992426d3146a4c098193706997a2e2a5 P e2f1b9f64c4d4917a133f18b471fab50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:44:52.707424 3608 ts_tablet_manager.cc:1905] T 992426d3146a4c098193706997a2e2a5 P e2f1b9f64c4d4917a133f18b471fab50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:44:52.710832 3608 ts_tablet_manager.cc:1918] T 992426d3146a4c098193706997a2e2a5 P e2f1b9f64c4d4917a133f18b471fab50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:44:52.710912 3608 log.cc:1196] T 992426d3146a4c098193706997a2e2a5 P e2f1b9f64c4d4917a133f18b471fab50: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-0/wal/wals/992426d3146a4c098193706997a2e2a5
I20260926 16:44:52.711182 3608 ts_tablet_manager.cc:1939] T 992426d3146a4c098193706997a2e2a5 P e2f1b9f64c4d4917a133f18b471fab50: Deleting consensus metadata
I20260926 16:44:52.712990 3395 catalog_manager.cc:4841] TS e2f1b9f64c4d4917a133f18b471fab50 (127.21.12.65:41567): tablet 992426d3146a4c098193706997a2e2a5 (table test-table [id=e9f65566d8b34f40b3ef0d1e21df5bc9]) successfully deleted
W20260926 16:44:52.791205 3604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:52.791491 3604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:52.791574 3604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:52.796273 3604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:52.796357 3604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:52.796458 3604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.66
I20260926 16:44:52.801620 3604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:38015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab
--principal=kudu/127.21.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.66:39565
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.66
--webserver_port=41773
--tserver_master_addrs=127.21.12.126:43623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.66
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:44:52.802704 3604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:52.803741 3604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:52.813426 3616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:52.813436 3613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:52.813577 3614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:52.813824 3604 server_base.cc:1034] running on GCE node
I20260926 16:44:52.814213 3604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:52.814857 3604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:52.816033 3604 hybrid_clock.cc:648] HybridClock initialized: now 1790441092816016 us; error 39 us; skew 500 ppm
Sep 26 16:44:52 dist-test-slave-n1hw krb5kdc[2822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441092, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:52.818780 3604 init.cc:377] Logged in from keytab as kudu/127.21.12.66@KRBTEST.COM (short username kudu)
I20260926 16:44:52.819664 3604 webserver.cc:460] Webserver started at http://127.21.12.66:41773/ using document root <none> and password file <none>
I20260926 16:44:52.820209 3604 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:52.820309 3604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:52.824010 3604 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:44:52.826081 3623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:52.827250 3604 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 16:44:52.827384 3604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-1/wal
uuid: "9e58b0c44a5b4076b135b9a2284d2335"
format_stamp: "Formatted at 2026-09-26 16:44:51 on dist-test-slave-n1hw"
server_key: "9bd2015b96e07997b162b17de979e025"
server_key_iv: "0937d75a70f0f08333df9ae20125b214"
server_key_version: "encryptionkey@0"
I20260926 16:44:52.827903 3604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:52.840215 3604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:52.843146 3604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:52.843339 3604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:52.843892 3604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:44:52.844713 3604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:44:52.844779 3604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:52.844849 3604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:44:52.844892 3604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:52.854110 3604 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.66:39565
I20260926 16:44:52.854136 3737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.66:39565 every 8 connection(s)
I20260926 16:44:52.855134 3604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
Sep 26 16:44:52 dist-test-slave-n1hw krb5kdc[2822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441092, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.66@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:44:52.864950 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 3604
I20260926 16:44:52.865331 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.67:46679
--local_ip_for_outbound_sockets=127.21.12.67
--tserver_master_addrs=127.21.12.126:43623
--webserver_port=41193
--webserver_interface=127.21.12.67
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab
--principal=kudu/127.21.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:38015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:44:52.869092 3602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:52.856960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:55457 (local address 127.21.12.126:43623)
0926 16:44:52.857096 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:52.857100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:52.858006 (+ 906us) server_negotiation.cc:407] Connection header received
0926 16:44:52.859400 (+ 1394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:52.859404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:52.859465 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:52.859571 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:52.860705 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:52.861244 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:52.862403 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:52.862652 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:52.862760 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:52.865450 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:52.865459 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:52.865462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:52.865530 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:52.866991 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:52.867503 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:52.867508 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:52.867510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:52.867587 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:52.867982 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:52.867986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:52.867988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:52.868178 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:52.868263 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:52.868865 (+ 602us) server_negotiation.cc:299] Negotiation successful
0926 16:44:52.868974 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 16:44:52.870398 3740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:52.857372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.66:55457)
0926 16:44:52.857868 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:52.857909 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:52.859190 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:52.859746 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:52.859753 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:52.860088 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:52.860551 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:52.860564 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:52.861377 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:52.861381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:52.862264 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:52.862274 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:52.862809 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:52.862813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:52.862972 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:52.863559 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:52.863574 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:52.865268 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:52.867111 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:52.867116 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:52.867131 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:52.867389 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:52.867687 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:52.867691 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:52.867692 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:52.867854 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:52.868292 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:52.868301 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:52.868615 (+ 314us) client_negotiation.cc:769] Sending connection context
0926 16:44:52.868826 (+ 211us) client_negotiation.cc:240] Negotiation successful
0926 16:44:52.869189 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":125,"threads_started":1}
I20260926 16:44:52.871613 3738 heartbeater.cc:347] Connected to a master server at 127.21.12.126:43623
I20260926 16:44:52.871876 3738 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:52.872496 3738 heartbeater.cc:511] Master 127.21.12.126:43623 requested a full tablet report, sending...
I20260926 16:44:52.873648 3408 ts_manager.cc:194] Registered new tserver with Master: 9e58b0c44a5b4076b135b9a2284d2335 (127.21.12.66:39565)
I20260926 16:44:52.874249 3408 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.66@KRBTEST.COM'} at 127.21.12.66:55457
W20260926 16:44:52.977392 3741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:52.977692 3741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:52.977813 3741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:52.981735 3741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:52.981874 3741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:52.981972 3741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.67
I20260926 16:44:52.987442 3741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:38015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab
--principal=kudu/127.21.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.67:46679
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.67
--webserver_port=41193
--tserver_master_addrs=127.21.12.126:43623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.67
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:44:52.988718 3741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:52.989964 3741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:52.999456 3746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:52.999456 3747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:52.999501 3749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:52.999943 3741 server_base.cc:1034] running on GCE node
I20260926 16:44:53.000288 3741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:53.000873 3741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:53.002053 3741 hybrid_clock.cc:648] HybridClock initialized: now 1790441093002030 us; error 38 us; skew 500 ppm
Sep 26 16:44:53 dist-test-slave-n1hw krb5kdc[2822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441093, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:53.004925 3741 init.cc:377] Logged in from keytab as kudu/127.21.12.67@KRBTEST.COM (short username kudu)
I20260926 16:44:53.005703 3741 webserver.cc:460] Webserver started at http://127.21.12.67:41193/ using document root <none> and password file <none>
I20260926 16:44:53.006340 3741 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:53.006451 3741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:53.010245 3741 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:44:53.012267 3756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:53.013545 3741 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:44:53.013707 3741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-2/wal
uuid: "cfd6bcea69494c108b9e46a933425537"
format_stamp: "Formatted at 2026-09-26 16:44:51 on dist-test-slave-n1hw"
server_key: "1f51aea165f163a1d2e3d21f569c8c54"
server_key_iv: "880a5223aa2074af0da259943003433b"
server_key_version: "encryptionkey@0"
I20260926 16:44:53.014284 3741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:53.029928 3741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:53.033150 3741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:53.033346 3741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:53.033968 3741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:44:53.034742 3741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:44:53.034811 3741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:53.034879 3741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:44:53.034921 3741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:53.043977 3741 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.67:46679
I20260926 16:44:53.044061 3870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.67:46679 every 8 connection(s)
I20260926 16:44:53.044939 3741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
I20260926 16:44:53.052209 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 3741
Sep 26 16:44:53 dist-test-slave-n1hw krb5kdc[2822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441093, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.67@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:44:53.059489 3602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:53.046865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:34905 (local address 127.21.12.126:43623)
0926 16:44:53.046999 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:53.047003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:53.047715 (+ 712us) server_negotiation.cc:407] Connection header received
0926 16:44:53.049093 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:53.049097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:53.049157 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:53.049259 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:53.050501 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:53.051032 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:53.052200 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:53.052478 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:53.052595 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:53.055227 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:53.055232 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:53.055235 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:53.055263 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:53.056938 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:53.057635 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:53.057638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:53.057639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:53.057725 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:53.058176 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:53.058181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:53.058183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:53.058417 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:53.058518 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:53.059108 (+ 590us) server_negotiation.cc:299] Negotiation successful
0926 16:44:53.059304 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 16:44:53.060380 3873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:53.047133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.67:34905)
0926 16:44:53.047560 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:53.047602 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:53.048872 (+ 1270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:53.049447 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:53.049455 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:53.049873 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:53.050351 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:53.050364 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:53.051165 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:53.051168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:53.052028 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:53.052037 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:53.052632 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:53.052637 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:53.052855 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:53.053501 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:53.053520 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:53.055078 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:53.057080 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:53.057087 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:53.057104 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:53.057483 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:53.057867 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:53.057870 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:53.057872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:53.058056 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:53.058535 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:53.058543 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:53.058824 (+ 281us) client_negotiation.cc:769] Sending connection context
0926 16:44:53.059043 (+ 219us) client_negotiation.cc:240] Negotiation successful
0926 16:44:53.059335 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":107,"threads_started":1}
I20260926 16:44:53.061492 3875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:53.052858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37882 (local address 127.21.12.126:43623)
0926 16:44:53.053134 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:53.053145 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:53.053190 (+ 45us) server_negotiation.cc:407] Connection header received
0926 16:44:53.053343 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:53.053346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:53.053411 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:53.053505 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:53.054380 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:53.054933 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:53.056160 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:53.056387 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:53.056495 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:53.058070 (+ 1575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:53.058073 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:53.058075 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:53.058096 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:53.059776 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:53.060192 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:53.060194 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:53.060195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:53.060261 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:53.060573 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:53.060577 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:53.060579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:53.060800 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:53.061024 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:53.061231 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 16:44:53.061352 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":95,"threads_started":1}
I20260926 16:44:53.061851 3871 heartbeater.cc:347] Connected to a master server at 127.21.12.126:43623
I20260926 16:44:53.062106 3871 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:53.062662 3871 heartbeater.cc:511] Master 127.21.12.126:43623 requested a full tablet report, sending...
I20260926 16:44:53.063715 3407 ts_manager.cc:194] Registered new tserver with Master: cfd6bcea69494c108b9e46a933425537 (127.21.12.67:46679)
I20260926 16:44:53.064283 3407 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.67@KRBTEST.COM'} at 127.21.12.67:34905
I20260926 16:44:53.075973 21553 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:44:53.080736 3875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:53.076652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37894 (local address 127.21.12.126:43623)
0926 16:44:53.076778 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:53.076781 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:53.077030 (+ 249us) server_negotiation.cc:407] Connection header received
0926 16:44:53.077186 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:53.077188 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:53.077289 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:53.077379 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:44:53.077631 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:53.078198 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:53.079132 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:53.079351 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:53.079432 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:53.079888 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:44:53.080302 (+ 414us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:44:53.080414 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:53.080547 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 16:44:53.080633 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 16:44:53.081460 3407 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:37894:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:44:53.084761 3407 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:44:53.092356 3531 tablet_service.cc:1467] Processing CreateTablet for tablet c05768e9aee64c00a88b13bf56a38340 (DEFAULT_TABLE table=test-table [id=dc81d1b64e7d4d2f87a0014315550658]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:44:53.093029 3531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c05768e9aee64c00a88b13bf56a38340. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:44:53.095942 3490 tablet_bootstrap.cc:492] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50: Bootstrap starting.
I20260926 16:44:53.096879 3490 tablet_bootstrap.cc:654] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50: Neither blocks nor log segments found. Creating new log.
I20260926 16:44:53.096936 3607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:53.091350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:39565 (local address 127.0.0.1:50034)
0926 16:44:53.091455 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:53.091471 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:53.091548 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:53.092623 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:53.092625 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:53.092643 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:44:53.092752 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:53.092758 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:53.094571 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:53.094575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:53.095468 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:53.095478 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:53.096492 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:53.096498 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:53.096598 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:53.096642 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 16:44:53.096718 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 16:44:53.096786 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":2}
I20260926 16:44:53.097046 3880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:53.091515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50034 (local address 127.21.12.66:39565)
0926 16:44:53.092058 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:53.092071 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:53.092094 (+ 23us) server_negotiation.cc:407] Connection header received
0926 16:44:53.092167 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:53.092172 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:53.092385 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:53.092907 (+ 522us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:44:53.093063 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:53.094414 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:53.095597 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:53.096341 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:53.096482 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:53.096634 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:53.096747 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 16:44:53.096870 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":202,"threads_started":1}
I20260926 16:44:53.098013 3881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:53.091953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.67:46679 (local address 127.0.0.1:47126)
0926 16:44:53.092195 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:53.092210 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:53.092298 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:53.093190 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:53.093193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:53.093213 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:44:53.093311 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:53.093317 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:53.094828 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:53.094833 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:53.095978 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:53.095991 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:53.097581 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:53.097585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:53.097677 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:53.097699 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 16:44:53.097807 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 16:44:53.097875 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":1,"thread_start_us":123,"threads_started":1}
I20260926 16:44:53.098837 3490 tablet_bootstrap.cc:492] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50: No bootstrap required, opened a new log
I20260926 16:44:53.098912 3490 ts_tablet_manager.cc:1397] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 16:44:53.099323 3490 raft_consensus.cc:348] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e58b0c44a5b4076b135b9a2284d2335" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 39565 } } peers { permanent_uuid: "cfd6bcea69494c108b9e46a933425537" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 46679 } } peers { permanent_uuid: "e2f1b9f64c4d4917a133f18b471fab50" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41567 } }
I20260926 16:44:53.099418 3490 raft_consensus.cc:374] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:44:53.099445 3490 raft_consensus.cc:729] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2f1b9f64c4d4917a133f18b471fab50, State: Initialized, Role: FOLLOWER
I20260926 16:44:53.099596 3490 consensus_queue.cc:260] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e58b0c44a5b4076b135b9a2284d2335" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 39565 } } peers { permanent_uuid: "cfd6bcea69494c108b9e46a933425537" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 46679 } } peers { permanent_uuid: "e2f1b9f64c4d4917a133f18b471fab50" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41567 } }
I20260926 16:44:53.099870 3490 ts_tablet_manager.cc:1428] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 16:44:53.099968 3671 tablet_service.cc:1467] Processing CreateTablet for tablet c05768e9aee64c00a88b13bf56a38340 (DEFAULT_TABLE table=test-table [id=dc81d1b64e7d4d2f87a0014315550658]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:44:53.100970 3671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c05768e9aee64c00a88b13bf56a38340. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:44:53.102178 3882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:53.092395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47126 (local address 127.21.12.67:46679)
0926 16:44:53.092735 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:53.092750 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:53.092768 (+ 18us) server_negotiation.cc:407] Connection header received
0926 16:44:53.092826 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:53.092830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:53.092973 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:53.093446 (+ 473us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:44:53.093626 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:53.094682 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:53.096194 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:53.097457 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:53.097580 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:53.097714 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:53.097838 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 16:44:53.102007 (+ 4169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":167,"threads_started":1}
I20260926 16:44:53.104995 3804 tablet_service.cc:1467] Processing CreateTablet for tablet c05768e9aee64c00a88b13bf56a38340 (DEFAULT_TABLE table=test-table [id=dc81d1b64e7d4d2f87a0014315550658]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:44:53.106041 3605 raft_consensus.cc:482] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:44:53.106069 3804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c05768e9aee64c00a88b13bf56a38340. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:44:53.106225 3605 raft_consensus.cc:504] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e58b0c44a5b4076b135b9a2284d2335" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 39565 } } peers { permanent_uuid: "cfd6bcea69494c108b9e46a933425537" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 46679 } } peers { permanent_uuid: "e2f1b9f64c4d4917a133f18b471fab50" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41567 } }
I20260926 16:44:53.107230 3605 leader_election.cc:288] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e58b0c44a5b4076b135b9a2284d2335 (127.21.12.66:39565), cfd6bcea69494c108b9e46a933425537 (127.21.12.67:46679)
I20260926 16:44:53.109712 3885 tablet_bootstrap.cc:492] T c05768e9aee64c00a88b13bf56a38340 P 9e58b0c44a5b4076b135b9a2284d2335: Bootstrap starting.
I20260926 16:44:53.112207 3601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:53.107365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:39565 (local address 127.21.12.65:60339)
0926 16:44:53.107591 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:53.107605 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:53.107685 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:53.108321 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:53.108323 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:53.108339 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:44:53.108437 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:53.108444 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:53.109788 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:53.109792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:53.110927 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:53.110935 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:53.111760 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:53.111764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:53.111887 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:53.111935 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 16:44:53.111987 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 16:44:53.112068 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":16}
I20260926 16:44:53.112421 3885 tablet_bootstrap.cc:654] T c05768e9aee64c00a88b13bf56a38340 P 9e58b0c44a5b4076b135b9a2284d2335: Neither blocks nor log segments found. Creating new log.
I20260926 16:44:53.112751 3880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:53.107849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:60339 (local address 127.21.12.66:39565)
0926 16:44:53.108012 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:53.108018 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:53.108030 (+ 12us) server_negotiation.cc:407] Connection header received
0926 16:44:53.108075 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:53.108078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:53.108137 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:53.108214 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:44:53.108533 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:53.109649 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:53.111053 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:53.111632 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:53.112490 (+ 858us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:53.112542 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:53.112577 (+ 35us) server_negotiation.cc:299] Negotiation successful
0926 16:44:53.112647 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260926 16:44:53.113291 3885 log.cc:824] T c05768e9aee64c00a88b13bf56a38340 P 9e58b0c44a5b4076b135b9a2284d2335: Log is configured to *not* fsync() on all Append() calls
I20260926 16:44:53.114102 3884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:53.107594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.67:46679 (local address 127.21.12.65:57121)
0926 16:44:53.108182 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:53.108198 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:53.108272 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:53.109139 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:53.109142 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:53.109160 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:44:53.109278 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:53.109284 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:53.110308 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:53.110312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:53.111931 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:53.111939 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:53.113665 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:53.113669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:53.113792 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:53.113818 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 16:44:53.113890 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 16:44:53.113956 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":24,"thread_start_us":94,"threads_started":1}
I20260926 16:44:53.114320 3886 tablet_bootstrap.cc:492] T c05768e9aee64c00a88b13bf56a38340 P cfd6bcea69494c108b9e46a933425537: Bootstrap starting.
I20260926 16:44:53.113209 3692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c05768e9aee64c00a88b13bf56a38340" candidate_uuid: "e2f1b9f64c4d4917a133f18b471fab50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e58b0c44a5b4076b135b9a2284d2335" is_pre_election: true
I20260926 16:44:53.114573 3882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:53.107962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:57121 (local address 127.21.12.67:46679)
0926 16:44:53.108715 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:53.108719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:53.108732 (+ 13us) server_negotiation.cc:407] Connection header received
0926 16:44:53.108775 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:53.108778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:53.108819 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:53.108889 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:44:53.109396 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:53.110167 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:53.112088 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:53.113442 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:53.113554 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:53.113596 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:53.114345 (+ 749us) server_negotiation.cc:299] Negotiation successful
0926 16:44:53.114465 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679}
W20260926 16:44:53.115224 3487 leader_election.cc:341] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9e58b0c44a5b4076b135b9a2284d2335 (127.21.12.66:39565): Illegal state: must be running to vote when last-logged opid is not known
I20260926 16:44:53.116295 3886 tablet_bootstrap.cc:654] T c05768e9aee64c00a88b13bf56a38340 P cfd6bcea69494c108b9e46a933425537: Neither blocks nor log segments found. Creating new log.
I20260926 16:44:53.115487 3825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c05768e9aee64c00a88b13bf56a38340" candidate_uuid: "e2f1b9f64c4d4917a133f18b471fab50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfd6bcea69494c108b9e46a933425537" is_pre_election: true
I20260926 16:44:53.117039 3886 log.cc:824] T c05768e9aee64c00a88b13bf56a38340 P cfd6bcea69494c108b9e46a933425537: Log is configured to *not* fsync() on all Append() calls
W20260926 16:44:53.117332 3484 leader_election.cc:341] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cfd6bcea69494c108b9e46a933425537 (127.21.12.67:46679): Illegal state: must be running to vote when last-logged opid is not known
I20260926 16:44:53.117431 3484 leader_election.cc:302] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2f1b9f64c4d4917a133f18b471fab50; no voters: 9e58b0c44a5b4076b135b9a2284d2335, cfd6bcea69494c108b9e46a933425537
I20260926 16:44:53.117729 3605 raft_consensus.cc:2726] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 16:44:53.118821 3885 tablet_bootstrap.cc:492] T c05768e9aee64c00a88b13bf56a38340 P 9e58b0c44a5b4076b135b9a2284d2335: No bootstrap required, opened a new log
I20260926 16:44:53.119004 3886 tablet_bootstrap.cc:492] T c05768e9aee64c00a88b13bf56a38340 P cfd6bcea69494c108b9e46a933425537: No bootstrap required, opened a new log
I20260926 16:44:53.119011 3885 ts_tablet_manager.cc:1397] T c05768e9aee64c00a88b13bf56a38340 P 9e58b0c44a5b4076b135b9a2284d2335: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260926 16:44:53.119153 3886 ts_tablet_manager.cc:1397] T c05768e9aee64c00a88b13bf56a38340 P cfd6bcea69494c108b9e46a933425537: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 16:44:53.121799 3885 raft_consensus.cc:348] T c05768e9aee64c00a88b13bf56a38340 P 9e58b0c44a5b4076b135b9a2284d2335 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e58b0c44a5b4076b135b9a2284d2335" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 39565 } } peers { permanent_uuid: "cfd6bcea69494c108b9e46a933425537" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 46679 } } peers { permanent_uuid: "e2f1b9f64c4d4917a133f18b471fab50" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41567 } }
I20260926 16:44:53.121799 3886 raft_consensus.cc:348] T c05768e9aee64c00a88b13bf56a38340 P cfd6bcea69494c108b9e46a933425537 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e58b0c44a5b4076b135b9a2284d2335" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 39565 } } peers { permanent_uuid: "cfd6bcea69494c108b9e46a933425537" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 46679 } } peers { permanent_uuid: "e2f1b9f64c4d4917a133f18b471fab50" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41567 } }
I20260926 16:44:53.122042 3886 raft_consensus.cc:374] T c05768e9aee64c00a88b13bf56a38340 P cfd6bcea69494c108b9e46a933425537 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:44:53.122042 3885 raft_consensus.cc:374] T c05768e9aee64c00a88b13bf56a38340 P 9e58b0c44a5b4076b135b9a2284d2335 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:44:53.122103 3886 raft_consensus.cc:729] T c05768e9aee64c00a88b13bf56a38340 P cfd6bcea69494c108b9e46a933425537 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfd6bcea69494c108b9e46a933425537, State: Initialized, Role: FOLLOWER
I20260926 16:44:53.122103 3885 raft_consensus.cc:729] T c05768e9aee64c00a88b13bf56a38340 P 9e58b0c44a5b4076b135b9a2284d2335 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e58b0c44a5b4076b135b9a2284d2335, State: Initialized, Role: FOLLOWER
I20260926 16:44:53.122555 3885 consensus_queue.cc:260] T c05768e9aee64c00a88b13bf56a38340 P 9e58b0c44a5b4076b135b9a2284d2335 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e58b0c44a5b4076b135b9a2284d2335" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 39565 } } peers { permanent_uuid: "cfd6bcea69494c108b9e46a933425537" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 46679 } } peers { permanent_uuid: "e2f1b9f64c4d4917a133f18b471fab50" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41567 } }
I20260926 16:44:53.122555 3886 consensus_queue.cc:260] T c05768e9aee64c00a88b13bf56a38340 P cfd6bcea69494c108b9e46a933425537 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e58b0c44a5b4076b135b9a2284d2335" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 39565 } } peers { permanent_uuid: "cfd6bcea69494c108b9e46a933425537" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 46679 } } peers { permanent_uuid: "e2f1b9f64c4d4917a133f18b471fab50" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41567 } }
I20260926 16:44:53.123183 3738 heartbeater.cc:503] Master 127.21.12.126:43623 was elected leader, sending a full tablet report...
I20260926 16:44:53.123423 3871 heartbeater.cc:503] Master 127.21.12.126:43623 was elected leader, sending a full tablet report...
I20260926 16:44:53.123535 3886 ts_tablet_manager.cc:1428] T c05768e9aee64c00a88b13bf56a38340 P cfd6bcea69494c108b9e46a933425537: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260926 16:44:53.123535 3885 ts_tablet_manager.cc:1428] T c05768e9aee64c00a88b13bf56a38340 P 9e58b0c44a5b4076b135b9a2284d2335: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260926 16:44:53.166075 3600 tablet.cc:2369] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:44:53.295881 3872 tablet.cc:2369] T c05768e9aee64c00a88b13bf56a38340 P cfd6bcea69494c108b9e46a933425537: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:44:53.346742 3889 raft_consensus.cc:482] T c05768e9aee64c00a88b13bf56a38340 P cfd6bcea69494c108b9e46a933425537 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:44:53.346956 3889 raft_consensus.cc:504] T c05768e9aee64c00a88b13bf56a38340 P cfd6bcea69494c108b9e46a933425537 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e58b0c44a5b4076b135b9a2284d2335" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 39565 } } peers { permanent_uuid: "cfd6bcea69494c108b9e46a933425537" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 46679 } } peers { permanent_uuid: "e2f1b9f64c4d4917a133f18b471fab50" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41567 } }
I20260926 16:44:53.348052 3889 leader_election.cc:288] T c05768e9aee64c00a88b13bf56a38340 P cfd6bcea69494c108b9e46a933425537 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e58b0c44a5b4076b135b9a2284d2335 (127.21.12.66:39565), e2f1b9f64c4d4917a133f18b471fab50 (127.21.12.65:41567)
I20260926 16:44:53.352239 3880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:53.348453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:59265 (local address 127.21.12.66:39565)
0926 16:44:53.348627 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:53.348631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:53.348647 (+ 16us) server_negotiation.cc:407] Connection header received
0926 16:44:53.348795 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:53.348798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:53.348851 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:53.348925 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:44:53.349350 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:53.350038 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:53.351172 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:53.351625 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:53.351718 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:53.351756 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:53.352025 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 16:44:53.352096 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260926 16:44:53.352267 3873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:53.348380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:39565 (local address 127.21.12.67:59265)
0926 16:44:53.348511 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:53.348540 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:53.348652 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:53.349046 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:53.349049 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:53.349097 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:44:53.349227 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:53.349233 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:53.350160 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:53.350163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:53.351049 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:53.351058 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:53.351741 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:53.351745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:53.351862 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:53.351910 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 16:44:53.351982 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 16:44:53.352114 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":69,"spinlock_wait_cycles":43520}
I20260926 16:44:53.352380 3892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:53.348771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:53545 (local address 127.21.12.65:41567)
0926 16:44:53.349073 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:53.349079 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:53.349091 (+ 12us) server_negotiation.cc:407] Connection header received
0926 16:44:53.349184 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:53.349188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:53.349238 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:53.349321 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:44:53.349568 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:53.350192 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:53.351267 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:53.351751 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:53.351866 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:53.352019 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:53.352175 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 16:44:53.352249 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":110,"threads_started":1}
I20260926 16:44:53.352486 3891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:53.348661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:41567 (local address 127.21.12.67:53545)
0926 16:44:53.348986 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:53.348997 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:53.349104 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:53.349333 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:53.349336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:53.349349 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:44:53.349480 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:53.349486 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:53.350305 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:53.350308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:53.351137 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:53.351148 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:53.351875 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:53.351877 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:53.352052 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:53.352075 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 16:44:53.352157 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 16:44:53.352218 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":22,"spinlock_wait_cycles":4096,"thread_start_us":135,"threads_started":1}
I20260926 16:44:53.352775 3692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c05768e9aee64c00a88b13bf56a38340" candidate_uuid: "cfd6bcea69494c108b9e46a933425537" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e58b0c44a5b4076b135b9a2284d2335" is_pre_election: true
I20260926 16:44:53.352980 3553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c05768e9aee64c00a88b13bf56a38340" candidate_uuid: "cfd6bcea69494c108b9e46a933425537" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f1b9f64c4d4917a133f18b471fab50" is_pre_election: true
I20260926 16:44:53.353062 3692 raft_consensus.cc:2454] T c05768e9aee64c00a88b13bf56a38340 P 9e58b0c44a5b4076b135b9a2284d2335 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfd6bcea69494c108b9e46a933425537 in term 0.
I20260926 16:44:53.353222 3553 raft_consensus.cc:2454] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfd6bcea69494c108b9e46a933425537 in term 0.
I20260926 16:44:53.353443 3760 leader_election.cc:302] T c05768e9aee64c00a88b13bf56a38340 P cfd6bcea69494c108b9e46a933425537 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e58b0c44a5b4076b135b9a2284d2335, cfd6bcea69494c108b9e46a933425537; no voters:
I20260926 16:44:53.353796 3889 raft_consensus.cc:2781] T c05768e9aee64c00a88b13bf56a38340 P cfd6bcea69494c108b9e46a933425537 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:44:53.353902 3889 raft_consensus.cc:482] T c05768e9aee64c00a88b13bf56a38340 P cfd6bcea69494c108b9e46a933425537 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:44:53.353948 3889 raft_consensus.cc:3037] T c05768e9aee64c00a88b13bf56a38340 P cfd6bcea69494c108b9e46a933425537 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:44:53.354951 3889 raft_consensus.cc:504] T c05768e9aee64c00a88b13bf56a38340 P cfd6bcea69494c108b9e46a933425537 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e58b0c44a5b4076b135b9a2284d2335" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 39565 } } peers { permanent_uuid: "cfd6bcea69494c108b9e46a933425537" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 46679 } } peers { permanent_uuid: "e2f1b9f64c4d4917a133f18b471fab50" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41567 } }
I20260926 16:44:53.355274 3889 leader_election.cc:288] T c05768e9aee64c00a88b13bf56a38340 P cfd6bcea69494c108b9e46a933425537 [CANDIDATE]: Term 1 election: Requested vote from peers 9e58b0c44a5b4076b135b9a2284d2335 (127.21.12.66:39565), e2f1b9f64c4d4917a133f18b471fab50 (127.21.12.65:41567)
I20260926 16:44:53.355593 3692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c05768e9aee64c00a88b13bf56a38340" candidate_uuid: "cfd6bcea69494c108b9e46a933425537" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e58b0c44a5b4076b135b9a2284d2335"
I20260926 16:44:53.355733 3692 raft_consensus.cc:3037] T c05768e9aee64c00a88b13bf56a38340 P 9e58b0c44a5b4076b135b9a2284d2335 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:44:53.355674 3553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c05768e9aee64c00a88b13bf56a38340" candidate_uuid: "cfd6bcea69494c108b9e46a933425537" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f1b9f64c4d4917a133f18b471fab50"
I20260926 16:44:53.355945 3553 raft_consensus.cc:3037] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [term 0 FOLLOWER]: Advancing to term 1
W20260926 16:44:53.356074 3739 tablet.cc:2369] T c05768e9aee64c00a88b13bf56a38340 P 9e58b0c44a5b4076b135b9a2284d2335: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:44:53.356950 3692 raft_consensus.cc:2454] T c05768e9aee64c00a88b13bf56a38340 P 9e58b0c44a5b4076b135b9a2284d2335 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfd6bcea69494c108b9e46a933425537 in term 1.
I20260926 16:44:53.356990 3553 raft_consensus.cc:2454] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfd6bcea69494c108b9e46a933425537 in term 1.
I20260926 16:44:53.357220 3760 leader_election.cc:302] T c05768e9aee64c00a88b13bf56a38340 P cfd6bcea69494c108b9e46a933425537 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e58b0c44a5b4076b135b9a2284d2335, cfd6bcea69494c108b9e46a933425537; no voters:
I20260926 16:44:53.357398 3889 raft_consensus.cc:2781] T c05768e9aee64c00a88b13bf56a38340 P cfd6bcea69494c108b9e46a933425537 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:44:53.357628 3889 raft_consensus.cc:686] T c05768e9aee64c00a88b13bf56a38340 P cfd6bcea69494c108b9e46a933425537 [term 1 LEADER]: Becoming Leader. State: Replica: cfd6bcea69494c108b9e46a933425537, State: Running, Role: LEADER
I20260926 16:44:53.357997 3889 consensus_queue.cc:237] T c05768e9aee64c00a88b13bf56a38340 P cfd6bcea69494c108b9e46a933425537 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e58b0c44a5b4076b135b9a2284d2335" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 39565 } } peers { permanent_uuid: "cfd6bcea69494c108b9e46a933425537" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 46679 } } peers { permanent_uuid: "e2f1b9f64c4d4917a133f18b471fab50" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41567 } }
I20260926 16:44:53.360919 3407 catalog_manager.cc:5495] T c05768e9aee64c00a88b13bf56a38340 P cfd6bcea69494c108b9e46a933425537 reported cstate change: term changed from 0 to 1, leader changed from <none> to cfd6bcea69494c108b9e46a933425537 (127.21.12.67). New cstate: current_term: 1 leader_uuid: "cfd6bcea69494c108b9e46a933425537" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e58b0c44a5b4076b135b9a2284d2335" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 39565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfd6bcea69494c108b9e46a933425537" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 46679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2f1b9f64c4d4917a133f18b471fab50" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41567 } health_report { overall_health: UNKNOWN } } }
I20260926 16:44:53.397102 3882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:53.393579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47136 (local address 127.21.12.67:46679)
0926 16:44:53.393735 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:53.393739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:53.393794 (+ 55us) server_negotiation.cc:407] Connection header received
0926 16:44:53.393934 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:53.393936 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:53.394005 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:53.394102 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:44:53.394305 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:53.394810 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:53.395644 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:53.395869 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:53.395960 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:53.396330 (+ 370us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:44:53.396756 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:44:53.396859 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:53.396936 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 16:44:53.396982 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 16:44:53.403007 3692 raft_consensus.cc:1261] T c05768e9aee64c00a88b13bf56a38340 P 9e58b0c44a5b4076b135b9a2284d2335 [term 1 FOLLOWER]: Refusing update from remote peer cfd6bcea69494c108b9e46a933425537: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:44:53.403038 3553 raft_consensus.cc:1261] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [term 1 FOLLOWER]: Refusing update from remote peer cfd6bcea69494c108b9e46a933425537: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:44:53.403782 3893 consensus_queue.cc:1035] T c05768e9aee64c00a88b13bf56a38340 P cfd6bcea69494c108b9e46a933425537 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2f1b9f64c4d4917a133f18b471fab50" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:44:53.404023 3889 consensus_queue.cc:1035] T c05768e9aee64c00a88b13bf56a38340 P cfd6bcea69494c108b9e46a933425537 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e58b0c44a5b4076b135b9a2284d2335" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 39565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:44:53.411922 3897 mvcc.cc:205] Tried to move back new op lower bound from 7333646718571102208 to 7333646718396456960. Current Snapshot: MvccSnapshot[applied={T|T < 7333646718571102208}]
I20260926 16:44:53.418761 3407 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:37894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:44:53.418936 3407 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:37894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:44:53.421424 3407 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17: Sending DeleteTablet for 3 replicas of tablet c05768e9aee64c00a88b13bf56a38340
I20260926 16:44:53.422204 3531 tablet_service.cc:1514] Processing DeleteTablet for tablet c05768e9aee64c00a88b13bf56a38340 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:44:53 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:52762
I20260926 16:44:53.422185 3671 tablet_service.cc:1514] Processing DeleteTablet for tablet c05768e9aee64c00a88b13bf56a38340 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:44:53 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:50034
I20260926 16:44:53.422441 3804 tablet_service.cc:1514] Processing DeleteTablet for tablet c05768e9aee64c00a88b13bf56a38340 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:44:53 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:47126
I20260926 16:44:53.422610 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 3447
I20260926 16:44:53.422631 3905 tablet_replica.cc:330] T c05768e9aee64c00a88b13bf56a38340 P 9e58b0c44a5b4076b135b9a2284d2335: stopping tablet replica
I20260926 16:44:53.422997 3905 raft_consensus.cc:2229] T c05768e9aee64c00a88b13bf56a38340 P 9e58b0c44a5b4076b135b9a2284d2335 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:44:53.423295 3905 raft_consensus.cc:2258] T c05768e9aee64c00a88b13bf56a38340 P 9e58b0c44a5b4076b135b9a2284d2335 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:44:53.424562 3906 tablet_replica.cc:330] T c05768e9aee64c00a88b13bf56a38340 P cfd6bcea69494c108b9e46a933425537: stopping tablet replica
I20260926 16:44:53.424975 3906 raft_consensus.cc:2229] T c05768e9aee64c00a88b13bf56a38340 P cfd6bcea69494c108b9e46a933425537 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:44:53.424984 3905 ts_tablet_manager.cc:1905] T c05768e9aee64c00a88b13bf56a38340 P 9e58b0c44a5b4076b135b9a2284d2335: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:44:53.425428 3906 raft_consensus.cc:2258] T c05768e9aee64c00a88b13bf56a38340 P cfd6bcea69494c108b9e46a933425537 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:44:53.427177 3906 ts_tablet_manager.cc:1905] T c05768e9aee64c00a88b13bf56a38340 P cfd6bcea69494c108b9e46a933425537: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:44:53.427769 3905 ts_tablet_manager.cc:1918] T c05768e9aee64c00a88b13bf56a38340 P 9e58b0c44a5b4076b135b9a2284d2335: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:44:53.427870 3905 log.cc:1196] T c05768e9aee64c00a88b13bf56a38340 P 9e58b0c44a5b4076b135b9a2284d2335: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-1/wal/wals/c05768e9aee64c00a88b13bf56a38340
I20260926 16:44:53.428210 3905 ts_tablet_manager.cc:1939] T c05768e9aee64c00a88b13bf56a38340 P 9e58b0c44a5b4076b135b9a2284d2335: Deleting consensus metadata
I20260926 16:44:53.429433 3395 catalog_manager.cc:4841] TS 9e58b0c44a5b4076b135b9a2284d2335 (127.21.12.66:39565): tablet c05768e9aee64c00a88b13bf56a38340 (table test-table [id=dc81d1b64e7d4d2f87a0014315550658]) successfully deleted
I20260926 16:44:53.431509 3906 ts_tablet_manager.cc:1918] T c05768e9aee64c00a88b13bf56a38340 P cfd6bcea69494c108b9e46a933425537: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:44:53.431697 3906 log.cc:1196] T c05768e9aee64c00a88b13bf56a38340 P cfd6bcea69494c108b9e46a933425537: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-2/wal/wals/c05768e9aee64c00a88b13bf56a38340
I20260926 16:44:53.432232 3906 ts_tablet_manager.cc:1939] T c05768e9aee64c00a88b13bf56a38340 P cfd6bcea69494c108b9e46a933425537: Deleting consensus metadata
I20260926 16:44:53.432649 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 3604
W20260926 16:44:53.434347 3395 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.12.65:41567 (error 108)
I20260926 16:44:53.434482 3392 catalog_manager.cc:4841] TS cfd6bcea69494c108b9e46a933425537 (127.21.12.67:46679): tablet c05768e9aee64c00a88b13bf56a38340 (table test-table [id=dc81d1b64e7d4d2f87a0014315550658]) successfully deleted
I20260926 16:44:53.435333 3881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:53.435071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:41567 (local address 127.0.0.1:52774)
0926 16:44:53.435206 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:53.435267 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.65:41567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260926 16:44:53.435524 3395 catalog_manager.cc:4584] TS e2f1b9f64c4d4917a133f18b471fab50 (127.21.12.65:41567): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c05768e9aee64c00a88b13bf56a38340: Network error: Client connection negotiation failed: client connection to 127.21.12.65:41567: connect: Connection refused (error 111)
I20260926 16:44:53.442428 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 3741
I20260926 16:44:53.448388 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 3374
I20260926 16:44:53.454903 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:43623
--webserver_interface=127.21.12.126
--webserver_port=42949
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:38015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:43623
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:44:53.562479 3908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:53.562750 3908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:53.562829 3908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:53.566787 3908 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:44:53.566861 3908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:53.566898 3908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:53.566955 3908 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:44:53.566993 3908 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:44:53.572036 3908 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:38015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:43623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:43623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.126
--webserver_port=42949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:44:53.572996 3908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:53.574043 3908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:53.582162 3916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:53.582201 3913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:53.582345 3908 server_base.cc:1034] running on GCE node
W20260926 16:44:53.582190 3914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:53.583315 3908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:53.584448 3908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:53.585642 3908 hybrid_clock.cc:648] HybridClock initialized: now 1790441093585624 us; error 54 us; skew 500 ppm
Sep 26 16:44:53 dist-test-slave-n1hw krb5kdc[2822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441093, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:53.588390 3908 init.cc:377] Logged in from keytab as kudu/127.21.12.126@KRBTEST.COM (short username kudu)
I20260926 16:44:53.589277 3908 webserver.cc:460] Webserver started at http://127.21.12.126:42949/ using document root <none> and password file <none>
I20260926 16:44:53.589870 3908 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:53.589960 3908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:53.593632 3908 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:44:53.595795 3923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:53.597016 3908 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:44:53.597162 3908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "e3ea10b58512425eb10bf0d7a55d3e17"
format_stamp: "Formatted at 2026-09-26 16:44:51 on dist-test-slave-n1hw"
server_key: "60ea5f920d55bd0c05a5067892acc098"
server_key_iv: "4b0e4304b593a2a1d6bf2f650ffcc8d2"
server_key_version: "encryptionkey@0"
I20260926 16:44:53.597684 3908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:53.617406 3908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:53.620718 3908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:53.620920 3908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:53.627662 3908 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.126:43623
I20260926 16:44:53.627952 3975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.126:43623 every 8 connection(s)
I20260926 16:44:53.628743 3908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:44:53.630745 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 3908
I20260926 16:44:53.631165 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.65:41567
--local_ip_for_outbound_sockets=127.21.12.65
--tserver_master_addrs=127.21.12.126:43623
--webserver_port=44711
--webserver_interface=127.21.12.65
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab
--principal=kudu/127.21.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:38015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:44:53.633598 3976 sys_catalog.cc:263] Verifying existing consensus state
I20260926 16:44:53.635254 3976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17: Bootstrap starting.
I20260926 16:44:53.652134 3976 log.cc:824] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17: Log is configured to *not* fsync() on all Append() calls
I20260926 16:44:53.663527 3976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 16:44:53.663909 3976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17: Bootstrap complete.
I20260926 16:44:53.667850 3976 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ea10b58512425eb10bf0d7a55d3e17" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 43623 } }
I20260926 16:44:53.668215 3976 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3ea10b58512425eb10bf0d7a55d3e17, State: Initialized, Role: FOLLOWER
I20260926 16:44:53.668727 3976 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ea10b58512425eb10bf0d7a55d3e17" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 43623 } }
I20260926 16:44:53.668844 3976 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:44:53.668926 3976 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:44:53.669040 3976 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [term 2 FOLLOWER]: Advancing to term 3
I20260926 16:44:53.671032 3976 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ea10b58512425eb10bf0d7a55d3e17" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 43623 } }
I20260926 16:44:53.671351 3976 leader_election.cc:302] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3ea10b58512425eb10bf0d7a55d3e17; no voters:
I20260926 16:44:53.671665 3976 leader_election.cc:288] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 16:44:53.671770 3980 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 16:44:53.671967 3980 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [term 3 LEADER]: Becoming Leader. State: Replica: e3ea10b58512425eb10bf0d7a55d3e17, State: Running, Role: LEADER
I20260926 16:44:53.672252 3980 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ea10b58512425eb10bf0d7a55d3e17" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 43623 } }
I20260926 16:44:53.673152 3980 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3ea10b58512425eb10bf0d7a55d3e17. Latest consensus state: current_term: 3 leader_uuid: "e3ea10b58512425eb10bf0d7a55d3e17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ea10b58512425eb10bf0d7a55d3e17" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 43623 } } }
I20260926 16:44:53.673302 3980 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [sys.catalog]: This master's current role is: LEADER
I20260926 16:44:53.673152 3976 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:44:53.673803 3981 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e3ea10b58512425eb10bf0d7a55d3e17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ea10b58512425eb10bf0d7a55d3e17" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 43623 } } }
I20260926 16:44:53.674010 3981 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3ea10b58512425eb10bf0d7a55d3e17 [sys.catalog]: This master's current role is: LEADER
I20260926 16:44:53.674674 3984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:44:53.679557 3984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:44:53.680219 3984 catalog_manager.cc:1259] Loaded cluster ID: 6acfdeb002a9424981a3f8091bcff7cc
I20260926 16:44:53.680276 3984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 16:44:53.681277 3984 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1f8b7bb9ad google::LogMessage::Fail() at ??:0
@ 0x7f1f8b7be88a google::LogMessage::SendToLog() at ??:0
@ 0x7f1f8b7bb4de google::LogMessage::Flush() at ??:0
@ 0x7f1f8b7bca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1f96100fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f1f96101d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f1f960fe579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f1f96142640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f1f8da0d3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1f8da0dc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f1f8da0f3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f1f8d9fe512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1f8f0466db start_thread at ??:0
@ 0x7f1f8a2b961f clone at ??:0
W20260926 16:44:53.768589 3978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:53.768810 3978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:53.768851 3978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:53.772889 3978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:53.772941 3978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:53.773027 3978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.65
I20260926 16:44:53.778522 3978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:38015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab
--principal=kudu/127.21.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.65:41567
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.65
--webserver_port=44711
--tserver_master_addrs=127.21.12.126:43623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.65
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:44:53.779510 3978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:53.780503 3978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:53.792399 4023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:53.792404 4019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:53.792719 3978 server_base.cc:1034] running on GCE node
W20260926 16:44:53.792510 4020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:53.793325 3978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:53.794056 3978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:53.795250 3978 hybrid_clock.cc:648] HybridClock initialized: now 1790441093795223 us; error 36 us; skew 500 ppm
Sep 26 16:44:53 dist-test-slave-n1hw krb5kdc[2822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441093, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:53.798249 3978 init.cc:377] Logged in from keytab as kudu/127.21.12.65@KRBTEST.COM (short username kudu)
I20260926 16:44:53.799319 3978 webserver.cc:460] Webserver started at http://127.21.12.65:44711/ using document root <none> and password file <none>
I20260926 16:44:53.800045 3978 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:53.800143 3978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:53.805696 3978 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:44:53.808557 4030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:53.810209 3978 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:44:53.810329 3978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-0/wal
uuid: "e2f1b9f64c4d4917a133f18b471fab50"
format_stamp: "Formatted at 2026-09-26 16:44:51 on dist-test-slave-n1hw"
server_key: "afaceace097be71f67d76240e609df8c"
server_key_iv: "5c7e68d2ff7d108238f7e847e8bc4e2a"
server_key_version: "encryptionkey@0"
I20260926 16:44:53.811017 3978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:53.833796 3978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:53.837528 3978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:53.837740 3978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:53.838568 3978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:44:53.839916 4037 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 16:44:53.844018 3978 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 16:44:53.844092 3978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.001s
I20260926 16:44:53.844156 3978 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 16:44:53.846403 3978 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 16:44:53.846472 3978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260926 16:44:53.846802 4037 tablet_bootstrap.cc:492] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50: Bootstrap starting.
I20260926 16:44:53.856793 3978 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.65:41567
I20260926 16:44:53.857972 3978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
I20260926 16:44:53.858423 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 3978
I20260926 16:44:53.858371 4146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.65:41567 every 8 connection(s)
I20260926 16:44:53.858942 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.66:39565
--local_ip_for_outbound_sockets=127.21.12.66
--tserver_master_addrs=127.21.12.126:43623
--webserver_port=41773
--webserver_interface=127.21.12.66
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab
--principal=kudu/127.21.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:38015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:44:53.867419 4037 log.cc:824] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50: Log is configured to *not* fsync() on all Append() calls
I20260926 16:44:53.874296 4037 tablet_bootstrap.cc:492] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 16:44:53.874698 4037 tablet_bootstrap.cc:492] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50: Bootstrap complete.
I20260926 16:44:53.876016 4037 ts_tablet_manager.cc:1397] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50: Time spent bootstrapping tablet: real 0.029s user 0.012s sys 0.014s
I20260926 16:44:53.878392 4037 raft_consensus.cc:348] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e58b0c44a5b4076b135b9a2284d2335" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 39565 } } peers { permanent_uuid: "cfd6bcea69494c108b9e46a933425537" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 46679 } } peers { permanent_uuid: "e2f1b9f64c4d4917a133f18b471fab50" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41567 } }
I20260926 16:44:53.878803 4037 raft_consensus.cc:729] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2f1b9f64c4d4917a133f18b471fab50, State: Initialized, Role: FOLLOWER
I20260926 16:44:53.879273 4037 consensus_queue.cc:260] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e58b0c44a5b4076b135b9a2284d2335" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 39565 } } peers { permanent_uuid: "cfd6bcea69494c108b9e46a933425537" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 46679 } } peers { permanent_uuid: "e2f1b9f64c4d4917a133f18b471fab50" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41567 } }
I20260926 16:44:53.880162 4037 ts_tablet_manager.cc:1428] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260926 16:44:53.926574 4150 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:53.860712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.65:38235)
0926 16:44:53.862042 (+ 1330us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:53.862111 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:53.863246 (+ 1135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:53.925628 (+ 62382us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1102,"thread_start_us":161,"threads_started":1}
W20260926 16:44:53.927150 4147 heartbeater.cc:644] Failed to heartbeat to 127.21.12.126:43623 (0 consecutive failures): Network error: Failed to ping master at 127.21.12.126:43623: Client connection negotiation failed: client connection to 127.21.12.126:43623: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 16:44:53.927888 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:53.927616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.65:34563)
0926 16:44:53.927753 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:53.927826 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 16:44:53.928550 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:53.928342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.65:48613)
0926 16:44:53.928447 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:53.928496 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 16:44:53.928718 4147 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 16:44:53.990585 4149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:53.990818 4149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:53.990859 4149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:53.994961 4149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:53.995028 4149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:53.995121 4149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.66
I20260926 16:44:54.000360 4149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:38015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab
--principal=kudu/127.21.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.66:39565
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.66
--webserver_port=41773
--tserver_master_addrs=127.21.12.126:43623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.66
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:44:54.001372 4149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:54.002393 4149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:54.011755 4157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:54.011770 4160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:54.011821 4149 server_base.cc:1034] running on GCE node
W20260926 16:44:54.011770 4158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:54.012367 4149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:54.012969 4149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:54.014161 4149 hybrid_clock.cc:648] HybridClock initialized: now 1790441094014144 us; error 28 us; skew 500 ppm
Sep 26 16:44:54 dist-test-slave-n1hw krb5kdc[2822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441094, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:54.017084 4149 init.cc:377] Logged in from keytab as kudu/127.21.12.66@KRBTEST.COM (short username kudu)
I20260926 16:44:54.017968 4149 webserver.cc:460] Webserver started at http://127.21.12.66:41773/ using document root <none> and password file <none>
I20260926 16:44:54.018471 4149 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:54.018536 4149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:54.022164 4149 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260926 16:44:54.024083 4167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:54.025278 4149 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:44:54.025394 4149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-1/wal
uuid: "9e58b0c44a5b4076b135b9a2284d2335"
format_stamp: "Formatted at 2026-09-26 16:44:51 on dist-test-slave-n1hw"
server_key: "9bd2015b96e07997b162b17de979e025"
server_key_iv: "0937d75a70f0f08333df9ae20125b214"
server_key_version: "encryptionkey@0"
I20260926 16:44:54.026032 4149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:54.054893 4149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:54.058327 4149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:54.058539 4149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:54.059137 4149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:44:54.059949 4149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:44:54.060010 4149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:54.060110 4149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:44:54.060149 4149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:54.068941 4149 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.66:39565
I20260926 16:44:54.069044 4281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.66:39565 every 8 connection(s)
I20260926 16:44:54.069984 4149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
I20260926 16:44:54.073822 4284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:54.072065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.66:39573)
0926 16:44:54.072514 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:54.072625 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":113,"threads_started":1}
W20260926 16:44:54.074267 4282 heartbeater.cc:644] Failed to heartbeat to 127.21.12.126:43623 (0 consecutive failures): Network error: Failed to ping master at 127.21.12.126:43623: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
I20260926 16:44:54.074834 4284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:54.074601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.66:46915)
0926 16:44:54.074698 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:54.074741 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 16:44:54.075472 4284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:54.075304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.66:39667)
0926 16:44:54.075390 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:54.075429 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 16:44:54.075682 4282 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 16:44:54.078555 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 4149
I20260926 16:44:54.078907 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.67:46679
--local_ip_for_outbound_sockets=127.21.12.67
--tserver_master_addrs=127.21.12.126:43623
--webserver_port=41193
--webserver_interface=127.21.12.67
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab
--principal=kudu/127.21.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:38015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:44:54.184876 4285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:54.185108 4285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:54.185148 4285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:54.189198 4285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:54.189247 4285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:54.189343 4285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.67
I20260926 16:44:54.194955 4285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:38015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab
--principal=kudu/127.21.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.67:46679
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.67
--webserver_port=41193
--tserver_master_addrs=127.21.12.126:43623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.67
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:44:54.195986 4285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:54.196972 4285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:54.206190 4290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:54.206195 4291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:54.206202 4293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:54.206446 4285 server_base.cc:1034] running on GCE node
I20260926 16:44:54.206957 4285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:54.207635 4285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:54.208822 4285 hybrid_clock.cc:648] HybridClock initialized: now 1790441094208799 us; error 40 us; skew 500 ppm
Sep 26 16:44:54 dist-test-slave-n1hw krb5kdc[2822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441094, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:54.211449 4285 init.cc:377] Logged in from keytab as kudu/127.21.12.67@KRBTEST.COM (short username kudu)
I20260926 16:44:54.212296 4285 webserver.cc:460] Webserver started at http://127.21.12.67:41193/ using document root <none> and password file <none>
I20260926 16:44:54.212831 4285 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:54.212927 4285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:54.216617 4285 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:44:54.218677 4300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:54.219926 4285 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:44:54.220065 4285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-2/wal
uuid: "cfd6bcea69494c108b9e46a933425537"
format_stamp: "Formatted at 2026-09-26 16:44:51 on dist-test-slave-n1hw"
server_key: "1f51aea165f163a1d2e3d21f569c8c54"
server_key_iv: "880a5223aa2074af0da259943003433b"
server_key_version: "encryptionkey@0"
I20260926 16:44:54.220624 4285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:54.236451 4285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:54.239646 4285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:54.239837 4285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:54.240377 4285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:44:54.241153 4285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:44:54.241218 4285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:54.241286 4285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:44:54.241335 4285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:54.250198 4285 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.67:46679
I20260926 16:44:54.250226 4414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.67:46679 every 8 connection(s)
I20260926 16:44:54.251137 4285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
I20260926 16:44:54.254827 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 4285
I20260926 16:44:54.255018 4417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:54.253093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.67:47967)
0926 16:44:54.253535 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:54.253648 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":110,"threads_started":1}
W20260926 16:44:54.255458 4415 heartbeater.cc:644] Failed to heartbeat to 127.21.12.126:43623 (0 consecutive failures): Network error: Failed to ping master at 127.21.12.126:43623: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
I20260926 16:44:54.256079 4417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:54.255836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.67:56079)
0926 16:44:54.255984 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:54.256027 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260926 16:44:54.256656 4417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:54.256450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.67:50897)
0926 16:44:54.256541 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:54.256580 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 16:44:54.256826 4415 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 16:44:54.929682 4419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:54.929252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.65:33307)
0926 16:44:54.929508 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:54.929575 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":78,"threads_started":1}
I20260926 16:44:55.072947 4420 raft_consensus.cc:482] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:44:55.073176 4420 raft_consensus.cc:504] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e58b0c44a5b4076b135b9a2284d2335" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 39565 } } peers { permanent_uuid: "cfd6bcea69494c108b9e46a933425537" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 46679 } } peers { permanent_uuid: "e2f1b9f64c4d4917a133f18b471fab50" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41567 } }
I20260926 16:44:55.074290 4420 leader_election.cc:288] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9e58b0c44a5b4076b135b9a2284d2335 (127.21.12.66:39565), cfd6bcea69494c108b9e46a933425537 (127.21.12.67:46679)
I20260926 16:44:55.076566 4424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:55.076155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.66:41743)
0926 16:44:55.076382 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:55.076445 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":56,"threads_started":1}
Sep 26 16:44:55 dist-test-slave-n1hw krb5kdc[2822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441093, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for kudu/127.21.12.66@KRBTEST.COM
Sep 26 16:44:55 dist-test-slave-n1hw krb5kdc[2822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441093, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for kudu/127.21.12.67@KRBTEST.COM
I20260926 16:44:55.087846 4419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:55.074455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:39565 (local address 127.21.12.65:34869)
0926 16:44:55.074662 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:55.074680 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:55.074777 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:55.076510 (+ 1733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:55.076515 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:55.077088 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:55.077536 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:55.077549 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:55.078851 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:55.078855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:55.079705 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:55.079715 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:55.080399 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:55.080402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:55.080609 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:55.081221 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:55.081237 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:55.082826 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:55.085319 (+ 2493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:55.085327 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:55.085343 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:55.085691 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:55.086070 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:55.086073 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:55.086075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:55.086242 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:55.086865 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:55.086871 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:55.087088 (+ 217us) client_negotiation.cc:769] Sending connection context
0926 16:44:55.087268 (+ 180us) client_negotiation.cc:240] Negotiation successful
0926 16:44:55.087597 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":147,"spinlock_wait_cycles":4352}
I20260926 16:44:55.087875 4421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:55.074506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.67:46679 (local address 127.21.12.65:50165)
0926 16:44:55.074783 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:55.074795 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:55.074869 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:55.076521 (+ 1652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:55.076524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:55.077085 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:55.077537 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:55.077549 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:55.078864 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:55.078867 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:55.079702 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:55.079711 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:55.080388 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:55.080392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:55.080605 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:55.081276 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:55.081288 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:55.083176 (+ 1888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:55.085572 (+ 2396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:55.085574 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:55.085576 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:55.085829 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:55.086205 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:55.086208 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:55.086209 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:55.086280 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:55.086877 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:55.086879 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:55.087087 (+ 208us) client_negotiation.cc:769] Sending connection context
0926 16:44:55.087260 (+ 173us) client_negotiation.cc:240] Negotiation successful
0926 16:44:55.087681 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":213,"spinlock_wait_cycles":23296,"thread_start_us":68,"threads_started":1}
I20260926 16:44:55.087950 4422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:55.074787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:50165 (local address 127.21.12.67:46679)
0926 16:44:55.075123 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:55.075138 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:55.075198 (+ 60us) server_negotiation.cc:407] Connection header received
0926 16:44:55.075829 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:55.075835 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:55.076261 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:55.076738 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:55.077704 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:55.078716 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:55.079821 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:55.080269 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:55.080434 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:55.083386 (+ 2952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:55.083392 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:55.083408 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:55.083446 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:55.085463 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:55.085933 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:55.085938 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:55.085942 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:55.086088 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:55.086389 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:55.086392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:55.086393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:55.086749 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:55.086893 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:55.087511 (+ 618us) server_negotiation.cc:299] Negotiation successful
0926 16:44:55.087779 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":60,"threads_started":1}
I20260926 16:44:55.087862 4423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:55.074779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:34869 (local address 127.21.12.66:39565)
0926 16:44:55.075111 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:55.075122 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:55.075199 (+ 77us) server_negotiation.cc:407] Connection header received
0926 16:44:55.075831 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:55.075839 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:55.076177 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:55.076471 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:55.077709 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:55.078702 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:55.079839 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:55.080269 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:55.080434 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:55.083037 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:55.083042 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:55.083053 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:55.083077 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:55.085200 (+ 2123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:55.085824 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:55.085830 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:55.085835 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:55.085980 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:55.086339 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:55.086345 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:55.086349 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:55.086746 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:55.086876 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:55.087405 (+ 529us) server_negotiation.cc:299] Negotiation successful
0926 16:44:55.087680 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":73,"threads_started":1}
W20260926 16:44:55.090593 4031 leader_election.cc:341] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cfd6bcea69494c108b9e46a933425537 (127.21.12.67:46679): Not found: Tablet not found: c05768e9aee64c00a88b13bf56a38340
W20260926 16:44:55.090837 4034 leader_election.cc:341] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9e58b0c44a5b4076b135b9a2284d2335 (127.21.12.66:39565): Not found: Tablet not found: c05768e9aee64c00a88b13bf56a38340
I20260926 16:44:55.090894 4034 leader_election.cc:302] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2f1b9f64c4d4917a133f18b471fab50; no voters: 9e58b0c44a5b4076b135b9a2284d2335, cfd6bcea69494c108b9e46a933425537
I20260926 16:44:55.091159 4420 raft_consensus.cc:2726] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 16:44:55.257738 4425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:55.257335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.67:53319)
0926 16:44:55.257571 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:55.257621 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":69,"threads_started":1}
I20260926 16:44:55.930900 4426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:55.930488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.65:44829)
0926 16:44:55.930743 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:55.930806 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":71,"threads_started":1}
I20260926 16:44:56.077669 4427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:56.077230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.66:55443)
0926 16:44:56.077495 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:56.077588 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":87,"threads_started":1}
I20260926 16:44:56.258911 4428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:56.258511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.67:47741)
0926 16:44:56.258756 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:56.258812 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":86,"threads_started":1}
I20260926 16:44:56.931989 4429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:56.931586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.65:46785)
0926 16:44:56.931835 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:56.931894 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":74,"threads_started":1}
I20260926 16:44:57.049450 4430 raft_consensus.cc:482] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:44:57.049582 4430 raft_consensus.cc:504] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e58b0c44a5b4076b135b9a2284d2335" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 39565 } } peers { permanent_uuid: "cfd6bcea69494c108b9e46a933425537" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 46679 } } peers { permanent_uuid: "e2f1b9f64c4d4917a133f18b471fab50" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41567 } }
I20260926 16:44:57.050159 4430 leader_election.cc:288] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9e58b0c44a5b4076b135b9a2284d2335 (127.21.12.66:39565), cfd6bcea69494c108b9e46a933425537 (127.21.12.67:46679)
W20260926 16:44:57.050964 4034 leader_election.cc:341] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9e58b0c44a5b4076b135b9a2284d2335 (127.21.12.66:39565): Not found: Tablet not found: c05768e9aee64c00a88b13bf56a38340
W20260926 16:44:57.051079 4031 leader_election.cc:341] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cfd6bcea69494c108b9e46a933425537 (127.21.12.67:46679): Not found: Tablet not found: c05768e9aee64c00a88b13bf56a38340
I20260926 16:44:57.051213 4031 leader_election.cc:302] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2f1b9f64c4d4917a133f18b471fab50; no voters: 9e58b0c44a5b4076b135b9a2284d2335, cfd6bcea69494c108b9e46a933425537
I20260926 16:44:57.051390 4430 raft_consensus.cc:2726] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 16:44:57.078954 4431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:57.078529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.66:46479)
0926 16:44:57.078792 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:57.078848 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":73,"threads_started":1}
I20260926 16:44:57.260087 4432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:57.259710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.67:41601)
0926 16:44:57.259954 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:57.260004 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":81,"threads_started":1}
I20260926 16:44:57.933221 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:57.932807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.65:46917)
0926 16:44:57.933061 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:57.933123 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":79,"threads_started":1}
I20260926 16:44:58.080186 4434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:58.079781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.66:57517)
0926 16:44:58.080046 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:58.080100 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":71,"threads_started":1}
I20260926 16:44:58.261165 4435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:58.260771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.67:47837)
0926 16:44:58.261016 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:58.261068 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":71,"threads_started":1}
I20260926 16:44:58.934643 4436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:58.934170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.65:53613)
0926 16:44:58.934453 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:58.934510 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":69,"threads_started":1}
I20260926 16:44:59.081431 4437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:59.080964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.66:45911)
0926 16:44:59.081250 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:59.081332 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":73,"threads_started":1}
I20260926 16:44:59.262357 4438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:59.261934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.67:40561)
0926 16:44:59.262208 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:59.262265 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":80,"threads_started":1}
I20260926 16:44:59.935895 4439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:59.935468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.65:38465)
0926 16:44:59.935732 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:59.935796 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":85,"threads_started":1}
I20260926 16:45:00.067878 4440 raft_consensus.cc:482] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:45:00.068009 4440 raft_consensus.cc:504] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e58b0c44a5b4076b135b9a2284d2335" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 39565 } } peers { permanent_uuid: "cfd6bcea69494c108b9e46a933425537" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 46679 } } peers { permanent_uuid: "e2f1b9f64c4d4917a133f18b471fab50" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41567 } }
I20260926 16:45:00.068583 4440 leader_election.cc:288] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9e58b0c44a5b4076b135b9a2284d2335 (127.21.12.66:39565), cfd6bcea69494c108b9e46a933425537 (127.21.12.67:46679)
W20260926 16:45:00.069355 4034 leader_election.cc:341] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9e58b0c44a5b4076b135b9a2284d2335 (127.21.12.66:39565): Not found: Tablet not found: c05768e9aee64c00a88b13bf56a38340
W20260926 16:45:00.069443 4031 leader_election.cc:341] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cfd6bcea69494c108b9e46a933425537 (127.21.12.67:46679): Not found: Tablet not found: c05768e9aee64c00a88b13bf56a38340
I20260926 16:45:00.069577 4031 leader_election.cc:302] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2f1b9f64c4d4917a133f18b471fab50; no voters: 9e58b0c44a5b4076b135b9a2284d2335, cfd6bcea69494c108b9e46a933425537
I20260926 16:45:00.069734 4440 raft_consensus.cc:2726] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 16:45:00.082607 4441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:00.082225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.66:43793)
0926 16:45:00.082465 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:00.082518 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":70,"threads_started":1}
I20260926 16:45:00.263573 4442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:00.263162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.67:57089)
0926 16:45:00.263423 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:00.263480 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":96,"threads_started":1}
I20260926 16:45:00.937197 4443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:00.936723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.65:48925)
0926 16:45:00.937003 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:00.937090 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":74,"threads_started":1}
I20260926 16:45:01.083804 4444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:01.083310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.66:50567)
0926 16:45:01.083667 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:01.083717 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":100,"threads_started":1}
I20260926 16:45:01.264814 4445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:01.264336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.67:48553)
0926 16:45:01.264668 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:01.264724 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":92,"threads_started":1}
I20260926 16:45:01.686132 4446 raft_consensus.cc:482] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:45:01.686288 4446 raft_consensus.cc:504] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e58b0c44a5b4076b135b9a2284d2335" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 39565 } } peers { permanent_uuid: "cfd6bcea69494c108b9e46a933425537" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 46679 } } peers { permanent_uuid: "e2f1b9f64c4d4917a133f18b471fab50" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41567 } }
I20260926 16:45:01.686836 4446 leader_election.cc:288] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9e58b0c44a5b4076b135b9a2284d2335 (127.21.12.66:39565), cfd6bcea69494c108b9e46a933425537 (127.21.12.67:46679)
W20260926 16:45:01.687621 4034 leader_election.cc:341] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9e58b0c44a5b4076b135b9a2284d2335 (127.21.12.66:39565): Not found: Tablet not found: c05768e9aee64c00a88b13bf56a38340
W20260926 16:45:01.687712 4031 leader_election.cc:341] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cfd6bcea69494c108b9e46a933425537 (127.21.12.67:46679): Not found: Tablet not found: c05768e9aee64c00a88b13bf56a38340
I20260926 16:45:01.687774 4031 leader_election.cc:302] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2f1b9f64c4d4917a133f18b471fab50; no voters: 9e58b0c44a5b4076b135b9a2284d2335, cfd6bcea69494c108b9e46a933425537
I20260926 16:45:01.687958 4446 raft_consensus.cc:2726] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 16:45:01.938598 4447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:01.938166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.65:53213)
0926 16:45:01.938441 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:01.938502 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":75,"threads_started":1}
I20260926 16:45:02.084867 4448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:02.084477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.66:59993)
0926 16:45:02.084725 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:02.084779 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":77,"threads_started":1}
I20260926 16:45:02.266036 4449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:02.265621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.67:43579)
0926 16:45:02.265877 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:02.265941 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":70,"threads_started":1}
I20260926 16:45:02.939802 4450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:02.939390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.65:59839)
0926 16:45:02.939644 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:02.939706 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":73,"threads_started":1}
I20260926 16:45:03.086100 4451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:03.085589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.66:37189)
0926 16:45:03.085913 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:03.086002 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":148,"threads_started":1}
I20260926 16:45:03.267197 4452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:03.266765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.67:46215)
0926 16:45:03.267048 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:03.267104 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":85,"threads_started":1}
I20260926 16:45:03.941119 4453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:03.940731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.65:36145)
0926 16:45:03.940969 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:03.941023 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":73,"threads_started":1}
I20260926 16:45:04.087339 4454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:04.086873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.66:53071)
0926 16:45:04.087139 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:04.087243 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":86,"threads_started":1}
I20260926 16:45:04.268338 4455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:04.267939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.67:59299)
0926 16:45:04.268194 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:04.268247 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":77,"threads_started":1}
I20260926 16:45:04.942356 4456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:04.941909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.65:44603)
0926 16:45:04.942204 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:04.942263 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":89,"threads_started":1}
I20260926 16:45:05.066339 4457 raft_consensus.cc:482] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:45:05.066496 4457 raft_consensus.cc:504] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e58b0c44a5b4076b135b9a2284d2335" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 39565 } } peers { permanent_uuid: "cfd6bcea69494c108b9e46a933425537" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 46679 } } peers { permanent_uuid: "e2f1b9f64c4d4917a133f18b471fab50" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 41567 } }
I20260926 16:45:05.066988 4457 leader_election.cc:288] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9e58b0c44a5b4076b135b9a2284d2335 (127.21.12.66:39565), cfd6bcea69494c108b9e46a933425537 (127.21.12.67:46679)
W20260926 16:45:05.067826 4031 leader_election.cc:341] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cfd6bcea69494c108b9e46a933425537 (127.21.12.67:46679): Not found: Tablet not found: c05768e9aee64c00a88b13bf56a38340
W20260926 16:45:05.067929 4034 leader_election.cc:341] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9e58b0c44a5b4076b135b9a2284d2335 (127.21.12.66:39565): Not found: Tablet not found: c05768e9aee64c00a88b13bf56a38340
I20260926 16:45:05.067996 4034 leader_election.cc:302] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2f1b9f64c4d4917a133f18b471fab50; no voters: 9e58b0c44a5b4076b135b9a2284d2335, cfd6bcea69494c108b9e46a933425537
I20260926 16:45:05.068135 4457 raft_consensus.cc:2726] T c05768e9aee64c00a88b13bf56a38340 P e2f1b9f64c4d4917a133f18b471fab50 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 16:45:05.088534 4458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:05.088140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.66:51689)
0926 16:45:05.088390 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:05.088444 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":73,"threads_started":1}
I20260926 16:45:05.269470 4459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:05.269086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.67:39493)
0926 16:45:05.269329 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:05.269380 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":87,"threads_started":1}
I20260926 16:45:05.943496 4460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:05.943099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.65:46675)
0926 16:45:05.943345 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:05.943402 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":72,"threads_started":1}
I20260926 16:45:06.089721 4461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:06.089308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.66:44031)
0926 16:45:06.089557 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:06.089614 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":75,"threads_started":1}
I20260926 16:45:06.270774 4462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:06.270330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.67:52853)
0926 16:45:06.270621 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:06.270675 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":87,"threads_started":1}
I20260926 16:45:06.944646 4463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:06.944239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.65:37641)
0926 16:45:06.944494 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:06.944552 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":93,"threads_started":1}
I20260926 16:45:07.091024 4464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:07.090621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.66:37035)
0926 16:45:07.090883 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:07.090936 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":101,"threads_started":1}
I20260926 16:45:07.271874 4465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:07.271487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.67:53205)
0926 16:45:07.271728 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:07.271782 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":79,"threads_started":1}
I20260926 16:45:07.945907 4466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:07.945409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.65:50507)
0926 16:45:07.945677 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:07.945737 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":70,"threads_started":1}
I20260926 16:45:08.092295 4467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:08.091879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.66:32831)
0926 16:45:08.092139 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:08.092196 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":74,"threads_started":1}
I20260926 16:45:08.272964 4468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:08.272569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.67:54997)
0926 16:45:08.272813 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:08.272869 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":68,"threads_started":1}
I20260926 16:45:08.947155 4469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:08.946764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.65:52239)
0926 16:45:08.947012 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:08.947063 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":85,"threads_started":1}
I20260926 16:45:09.093587 4470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:09.093193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43623 (local address 127.21.12.66:33485)
0926 16:45:09.093437 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:09.093494 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":81,"threads_started":1}
I20260926 16:45:09.256196 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 3978
I20260926 16:45:09.261646 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 4149
I20260926 16:45:09.266789 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 4285
2026-09-26T16:45:09Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19495 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:45:09 dist-test-slave-n1hw krb5kdc[4473](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:45:09 dist-test-slave-n1hw krb5kdc[4473](info): set up 2 sockets
Sep 26 16:45:09 dist-test-slave-n1hw krb5kdc[4473](info): commencing operation
krb5kdc: starting...
W20260926 16:45:10.332191 21553 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Sep 26 16:45:10 dist-test-slave-n1hw krb5kdc[4473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:45:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:45:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
I20260926 16:45:10.456774 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:35725
--webserver_interface=127.21.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:45327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:35725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:45:10.568754 4486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:45:10.569060 4486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:45:10.569134 4486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:45:10.573144 4486 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 16:45:10.573232 4486 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 16:45:10.573261 4486 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 16:45:10.573285 4486 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:45:10.573326 4486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:45:10.573374 4486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:45:10.573432 4486 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:45:10.573472 4486 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:45:10.578997 4486 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:45327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:35725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:35725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:45:10.580113 4486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:45:10.581175 4486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:45:10.589730 4494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:45:10.589843 4492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:45:10.589730 4491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:45:10.589927 4486 server_base.cc:1034] running on GCE node
I20260926 16:45:10.590715 4486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:45:10.592221 4486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:45:10.593367 4486 hybrid_clock.cc:648] HybridClock initialized: now 1790441110593339 us; error 42 us; skew 500 ppm
Sep 26 16:45:10 dist-test-slave-n1hw krb5kdc[4473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441110, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:45:10.597007 4486 init.cc:377] Logged in from keytab as kudu/127.21.12.126@KRBTEST.COM (short username kudu)
I20260926 16:45:10.597998 4486 webserver.cc:460] Webserver started at http://127.21.12.126:41287/ using document root <none> and password file <none>
I20260926 16:45:10.598569 4486 fs_manager.cc:359] Metadata directory not provided
I20260926 16:45:10.598672 4486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:45:10.598905 4486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:45:10.600919 4486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "25a40b6cfa6544cdb74879e3751924c4"
format_stamp: "Formatted at 2026-09-26 16:45:10 on dist-test-slave-n1hw"
server_key: "7a3a695de4ee7cd939009ecc2d9e3f4e"
server_key_iv: "c6ba6db48d53dbb31bc22985e3583c26"
server_key_version: "encryptionkey@0"
I20260926 16:45:10.601476 4486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "25a40b6cfa6544cdb74879e3751924c4"
format_stamp: "Formatted at 2026-09-26 16:45:10 on dist-test-slave-n1hw"
server_key: "7a3a695de4ee7cd939009ecc2d9e3f4e"
server_key_iv: "c6ba6db48d53dbb31bc22985e3583c26"
server_key_version: "encryptionkey@0"
I20260926 16:45:10.605279 4486 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 16:45:10.607852 4501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:45:10.609081 4486 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 16:45:10.609222 4486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "25a40b6cfa6544cdb74879e3751924c4"
format_stamp: "Formatted at 2026-09-26 16:45:10 on dist-test-slave-n1hw"
server_key: "7a3a695de4ee7cd939009ecc2d9e3f4e"
server_key_iv: "c6ba6db48d53dbb31bc22985e3583c26"
server_key_version: "encryptionkey@0"
I20260926 16:45:10.609332 4486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:45:10.625845 4486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:45:10.628656 4486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:45:10.628854 4486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:45:10.635577 4486 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.126:35725
I20260926 16:45:10.635900 4553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.126:35725 every 8 connection(s)
I20260926 16:45:10.636543 4486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:45:10.639428 4554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:45:10.642938 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 4486
I20260926 16:45:10.643090 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:45:10.643368 21553 external_mini_cluster.cc:1428] Setting key 50104377cec456f3132ab4e607b41564
I20260926 16:45:10.644898 4554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4: Bootstrap starting.
I20260926 16:45:10.647369 4554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4: Neither blocks nor log segments found. Creating new log.
I20260926 16:45:10.648083 4554 log.cc:824] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4: Log is configured to *not* fsync() on all Append() calls
I20260926 16:45:10.649960 4554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4: No bootstrap required, opened a new log
Sep 26 16:45:10 dist-test-slave-n1hw krb5kdc[4473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:45:10.653800 4554 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25a40b6cfa6544cdb74879e3751924c4" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 35725 } }
I20260926 16:45:10.654055 4554 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:45:10.654135 4554 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25a40b6cfa6544cdb74879e3751924c4, State: Initialized, Role: FOLLOWER
I20260926 16:45:10.654563 4554 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25a40b6cfa6544cdb74879e3751924c4" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 35725 } }
I20260926 16:45:10.654713 4554 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:45:10.654799 4554 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:45:10.654917 4554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:45:10.655789 4554 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25a40b6cfa6544cdb74879e3751924c4" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 35725 } }
I20260926 16:45:10.656124 4554 leader_election.cc:302] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25a40b6cfa6544cdb74879e3751924c4; no voters:
I20260926 16:45:10.656443 4554 leader_election.cc:288] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:45:10.656508 4559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:45:10.656751 4559 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [term 1 LEADER]: Becoming Leader. State: Replica: 25a40b6cfa6544cdb74879e3751924c4, State: Running, Role: LEADER
I20260926 16:45:10.657090 4559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25a40b6cfa6544cdb74879e3751924c4" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 35725 } }
I20260926 16:45:10.657651 4557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:10.644742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37146 (local address 127.21.12.126:35725)
0926 16:45:10.645130 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0926 16:45:10.645147 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 16:45:10.645180 (+ 33us) server_negotiation.cc:407] Connection header received
0926 16:45:10.645840 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:45:10.645858 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:45:10.646167 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:45:10.646428 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:45:10.647329 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:10.648441 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:10.649315 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:10.649845 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:10.650006 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:10.652432 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:45:10.652437 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:45:10.652449 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:45:10.652474 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:45:10.654457 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:45:10.655031 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:45:10.655037 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:45:10.655043 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:45:10.655167 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:45:10.655456 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:45:10.655461 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:45:10.655463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:45:10.655860 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:45:10.656062 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 16:45:10.656415 (+ 353us) server_negotiation.cc:299] Negotiation successful
0926 16:45:10.656768 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":125,"threads_started":1}
I20260926 16:45:10.657799 4554 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:45:10.658317 4559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25a40b6cfa6544cdb74879e3751924c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25a40b6cfa6544cdb74879e3751924c4" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 35725 } } }
I20260926 16:45:10.658501 4559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [sys.catalog]: This master's current role is: LEADER
I20260926 16:45:10.658708 4560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25a40b6cfa6544cdb74879e3751924c4. Latest consensus state: current_term: 1 leader_uuid: "25a40b6cfa6544cdb74879e3751924c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25a40b6cfa6544cdb74879e3751924c4" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 35725 } } }
I20260926 16:45:10.658808 4560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [sys.catalog]: This master's current role is: LEADER
I20260926 16:45:10.658957 4565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:45:10.662226 4565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:45:10.667870 4565 catalog_manager.cc:1347] Generated new cluster ID: 8099a53dc86f448086e3dea5e9ed0398
I20260926 16:45:10.667982 4565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:45:10.684113 4565 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:45:10.685087 4565 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:45:10.693281 4565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4: Generated new TSK 0
I20260926 16:45:10.694377 4565 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab.
Entry for principal kudu/127.21.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab.
I20260926 16:45:10.733078 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.65:0
--local_ip_for_outbound_sockets=127.21.12.65
--webserver_interface=127.21.12.65
--webserver_port=0
--tserver_master_addrs=127.21.12.126:35725
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab
--principal=kudu/127.21.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:45327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:45:10.841952 4581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:45:10.842193 4581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:45:10.842259 4581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:45:10.846166 4581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:45:10.846217 4581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:45:10.846325 4581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.65
I20260926 16:45:10.851471 4581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:45327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab
--principal=kudu/127.21.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.65
--webserver_port=0
--tserver_master_addrs=127.21.12.126:35725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.65
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:45:10.852443 4581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:45:10.853435 4581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:45:10.862567 4589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:45:10.862550 4586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:45:10.862658 4587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:45:10.862818 4581 server_base.cc:1034] running on GCE node
I20260926 16:45:10.863261 4581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:45:10.863870 4581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:45:10.865029 4581 hybrid_clock.cc:648] HybridClock initialized: now 1790441110864998 us; error 43 us; skew 500 ppm
Sep 26 16:45:10 dist-test-slave-n1hw krb5kdc[4473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441110, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:45:10.867643 4581 init.cc:377] Logged in from keytab as kudu/127.21.12.65@KRBTEST.COM (short username kudu)
I20260926 16:45:10.868546 4581 webserver.cc:460] Webserver started at http://127.21.12.65:32957/ using document root <none> and password file <none>
I20260926 16:45:10.869081 4581 fs_manager.cc:359] Metadata directory not provided
I20260926 16:45:10.869189 4581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:45:10.869415 4581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:45:10.871388 4581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-0/data/instance:
uuid: "9847782936414dd38e658a1faf3d2234"
format_stamp: "Formatted at 2026-09-26 16:45:10 on dist-test-slave-n1hw"
server_key: "f95bbc156153ccc503343b53f20644e4"
server_key_iv: "2ed0b0449f353e1bf084a77ddb634014"
server_key_version: "encryptionkey@0"
I20260926 16:45:10.871933 4581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance:
uuid: "9847782936414dd38e658a1faf3d2234"
format_stamp: "Formatted at 2026-09-26 16:45:10 on dist-test-slave-n1hw"
server_key: "f95bbc156153ccc503343b53f20644e4"
server_key_iv: "2ed0b0449f353e1bf084a77ddb634014"
server_key_version: "encryptionkey@0"
I20260926 16:45:10.875856 4581 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 16:45:10.878203 4596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:45:10.879478 4581 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:45:10.879570 4581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-0/wal
uuid: "9847782936414dd38e658a1faf3d2234"
format_stamp: "Formatted at 2026-09-26 16:45:10 on dist-test-slave-n1hw"
server_key: "f95bbc156153ccc503343b53f20644e4"
server_key_iv: "2ed0b0449f353e1bf084a77ddb634014"
server_key_version: "encryptionkey@0"
I20260926 16:45:10.879647 4581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:45:10.895211 4581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:45:10.898427 4581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:45:10.898574 4581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:45:10.899179 4581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:45:10.900090 4581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:45:10.900159 4581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:45:10.900218 4581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:45:10.900243 4581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:45:10.908762 4581 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.65:34405
I20260926 16:45:10.908784 4711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.65:34405 every 8 connection(s)
I20260926 16:45:10.909657 4581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
Sep 26 16:45:10 dist-test-slave-n1hw krb5kdc[4473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441110, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:45:10.919411 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 4581
I20260926 16:45:10.919550 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance
I20260926 16:45:10.919839 21553 external_mini_cluster.cc:1428] Setting key d371963f4b79e6ef291e1179d82c6ece
I20260926 16:45:10.923982 4557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:10.911569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:37877 (local address 127.21.12.126:35725)
0926 16:45:10.911743 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 16:45:10.911747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:45:10.912508 (+ 761us) server_negotiation.cc:407] Connection header received
0926 16:45:10.914045 (+ 1537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:45:10.914050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:45:10.914110 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:45:10.914192 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:45:10.915359 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:10.915973 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:10.917093 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:10.917339 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:10.917425 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:10.920068 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:45:10.920073 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:45:10.920076 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:45:10.920104 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:45:10.921724 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:45:10.922218 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:45:10.922225 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:45:10.922230 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:45:10.922326 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:45:10.922640 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:45:10.922647 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:45:10.922651 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:45:10.922900 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:45:10.923001 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 16:45:10.923643 (+ 642us) server_negotiation.cc:299] Negotiation successful
0926 16:45:10.923784 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 16:45:10.924768 4714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:10.911929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.65:37877)
0926 16:45:10.912357 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:10.912399 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 16:45:10.913738 (+ 1339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:45:10.914351 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:45:10.914359 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:45:10.914731 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:45:10.915199 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:10.915212 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:10.916098 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:10.916100 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:10.916958 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:10.916966 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:10.917469 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:10.917471 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:10.917608 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:10.918296 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:45:10.918312 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:45:10.919827 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:45:10.921862 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:45:10.921866 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:45:10.921877 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:45:10.922117 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:45:10.922424 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:45:10.922426 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:45:10.922427 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:45:10.922551 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:45:10.923079 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:45:10.923086 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:45:10.923394 (+ 308us) client_negotiation.cc:769] Sending connection context
0926 16:45:10.923554 (+ 160us) client_negotiation.cc:240] Negotiation successful
0926 16:45:10.923800 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":110,"threads_started":1}
I20260926 16:45:10.926000 4712 heartbeater.cc:347] Connected to a master server at 127.21.12.126:35725
I20260926 16:45:10.926249 4712 heartbeater.cc:464] Registering TS with master...
I20260926 16:45:10.926709 4712 heartbeater.cc:511] Master 127.21.12.126:35725 requested a full tablet report, sending...
I20260926 16:45:10.928128 4518 ts_manager.cc:194] Registered new tserver with Master: 9847782936414dd38e658a1faf3d2234 (127.21.12.65:34405)
I20260926 16:45:10.929221 4518 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.65@KRBTEST.COM'} at 127.21.12.65:37877
WARNING: no policy specified for kudu/127.21.12.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab.
Entry for principal kudu/127.21.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab.
I20260926 16:45:10.949092 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.66:0
--local_ip_for_outbound_sockets=127.21.12.66
--webserver_interface=127.21.12.66
--webserver_port=0
--tserver_master_addrs=127.21.12.126:35725
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab
--principal=kudu/127.21.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:45327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:45:11.059129 4717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:45:11.059392 4717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:45:11.059437 4717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:45:11.063534 4717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:45:11.063598 4717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:45:11.063666 4717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.66
I20260926 16:45:11.068890 4717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:45327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab
--principal=kudu/127.21.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.66
--webserver_port=0
--tserver_master_addrs=127.21.12.126:35725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.66
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:45:11.069955 4717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:45:11.071343 4717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:45:11.081059 4725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:45:11.081048 4722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:45:11.081085 4723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:45:11.081409 4717 server_base.cc:1034] running on GCE node
I20260926 16:45:11.081928 4717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:45:11.082695 4717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:45:11.083864 4717 hybrid_clock.cc:648] HybridClock initialized: now 1790441111083850 us; error 27 us; skew 500 ppm
Sep 26 16:45:11 dist-test-slave-n1hw krb5kdc[4473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441111, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:45:11.086858 4717 init.cc:377] Logged in from keytab as kudu/127.21.12.66@KRBTEST.COM (short username kudu)
I20260926 16:45:11.087705 4717 webserver.cc:460] Webserver started at http://127.21.12.66:44039/ using document root <none> and password file <none>
I20260926 16:45:11.088259 4717 fs_manager.cc:359] Metadata directory not provided
I20260926 16:45:11.088322 4717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:45:11.088532 4717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:45:11.090654 4717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-1/data/instance:
uuid: "763e6b4243744af68be266b9d09c1aa7"
format_stamp: "Formatted at 2026-09-26 16:45:11 on dist-test-slave-n1hw"
server_key: "d9f8d6c1f50a439755e30ec80df2fe0d"
server_key_iv: "6b6111300153a74f160696fde1c3cc38"
server_key_version: "encryptionkey@0"
I20260926 16:45:11.091202 4717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance:
uuid: "763e6b4243744af68be266b9d09c1aa7"
format_stamp: "Formatted at 2026-09-26 16:45:11 on dist-test-slave-n1hw"
server_key: "d9f8d6c1f50a439755e30ec80df2fe0d"
server_key_iv: "6b6111300153a74f160696fde1c3cc38"
server_key_version: "encryptionkey@0"
I20260926 16:45:11.094893 4717 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:45:11.097227 4732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:45:11.098462 4717 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:45:11.098588 4717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-1/wal
uuid: "763e6b4243744af68be266b9d09c1aa7"
format_stamp: "Formatted at 2026-09-26 16:45:11 on dist-test-slave-n1hw"
server_key: "d9f8d6c1f50a439755e30ec80df2fe0d"
server_key_iv: "6b6111300153a74f160696fde1c3cc38"
server_key_version: "encryptionkey@0"
I20260926 16:45:11.098690 4717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:45:11.108632 4717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:45:11.111397 4717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:45:11.111584 4717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:45:11.112176 4717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:45:11.112995 4717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:45:11.113041 4717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:45:11.113106 4717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:45:11.113183 4717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:45:11.121989 4717 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.66:35041
I20260926 16:45:11.122020 4846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.66:35041 every 8 connection(s)
I20260926 16:45:11.122840 4717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
I20260926 16:45:11.125334 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 4717
I20260926 16:45:11.125450 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance
I20260926 16:45:11.125771 21553 external_mini_cluster.cc:1428] Setting key f3d2fcebdf2069bd7fc924e227d8d427
Sep 26 16:45:11 dist-test-slave-n1hw krb5kdc[4473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441111, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.66@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.12.67@KRBTEST.COM; defaulting to no policy
I20260926 16:45:11.137666 4557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:11.124562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:38165 (local address 127.21.12.126:35725)
0926 16:45:11.124707 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 16:45:11.124712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:45:11.125740 (+ 1028us) server_negotiation.cc:407] Connection header received
0926 16:45:11.127333 (+ 1593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:45:11.127337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:45:11.127389 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:45:11.127469 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:45:11.128606 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:11.129443 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:11.130773 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:11.131258 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:11.131387 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:11.133844 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:45:11.133849 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:45:11.133855 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:45:11.133887 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:45:11.135547 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:45:11.136017 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:45:11.136020 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:45:11.136021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:45:11.136092 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:45:11.136511 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:45:11.136513 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:45:11.136514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:45:11.136683 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:45:11.136760 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 16:45:11.137441 (+ 681us) server_negotiation.cc:299] Negotiation successful
0926 16:45:11.137542 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.67@KRBTEST.COM" created.
I20260926 16:45:11.138646 4849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:11.125003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.66:38165)
0926 16:45:11.125590 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:11.125632 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 16:45:11.127122 (+ 1490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:45:11.127660 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:45:11.127667 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:45:11.128003 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:45:11.128467 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:11.128479 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:11.129594 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:11.129598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:11.130623 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:11.130633 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:11.131407 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:11.131410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:11.131562 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:11.132202 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:45:11.132219 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:45:11.133640 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:45:11.135659 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:45:11.135664 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:45:11.135676 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:45:11.135929 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:45:11.136260 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:45:11.136266 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:45:11.136271 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:45:11.136428 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:45:11.136789 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:45:11.136794 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:45:11.137021 (+ 227us) client_negotiation.cc:769] Sending connection context
0926 16:45:11.137383 (+ 362us) client_negotiation.cc:240] Negotiation successful
0926 16:45:11.137645 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":148,"threads_started":1}
I20260926 16:45:11.139889 4847 heartbeater.cc:347] Connected to a master server at 127.21.12.126:35725
I20260926 16:45:11.140156 4847 heartbeater.cc:464] Registering TS with master...
I20260926 16:45:11.140818 4847 heartbeater.cc:511] Master 127.21.12.126:35725 requested a full tablet report, sending...
I20260926 16:45:11.141938 4518 ts_manager.cc:194] Registered new tserver with Master: 763e6b4243744af68be266b9d09c1aa7 (127.21.12.66:35041)
I20260926 16:45:11.142555 4518 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.66@KRBTEST.COM'} at 127.21.12.66:38165
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab.
Entry for principal kudu/127.21.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab.
I20260926 16:45:11.154569 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.67:0
--local_ip_for_outbound_sockets=127.21.12.67
--webserver_interface=127.21.12.67
--webserver_port=0
--tserver_master_addrs=127.21.12.126:35725
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab
--principal=kudu/127.21.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:45327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:45:11.273247 4852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:45:11.273557 4852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:45:11.273648 4852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:45:11.280054 4852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:45:11.280162 4852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:45:11.280270 4852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.67
I20260926 16:45:11.289605 4852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:45327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab
--principal=kudu/127.21.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.67
--webserver_port=0
--tserver_master_addrs=127.21.12.126:35725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.67
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:45:11.291224 4852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:45:11.292695 4852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:45:11.303467 4857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:45:11.303452 4860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:45:11.303622 4852 server_base.cc:1034] running on GCE node
W20260926 16:45:11.303678 4858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:45:11.304116 4852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:45:11.304745 4852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:45:11.305888 4852 hybrid_clock.cc:648] HybridClock initialized: now 1790441111305878 us; error 24 us; skew 500 ppm
Sep 26 16:45:11 dist-test-slave-n1hw krb5kdc[4473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441111, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:45:11.308476 4852 init.cc:377] Logged in from keytab as kudu/127.21.12.67@KRBTEST.COM (short username kudu)
I20260926 16:45:11.309257 4852 webserver.cc:460] Webserver started at http://127.21.12.67:33883/ using document root <none> and password file <none>
I20260926 16:45:11.310016 4852 fs_manager.cc:359] Metadata directory not provided
I20260926 16:45:11.310163 4852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:45:11.310392 4852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:45:11.312537 4852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-2/data/instance:
uuid: "ed20749a2d644876849e5f81fe7bd137"
format_stamp: "Formatted at 2026-09-26 16:45:11 on dist-test-slave-n1hw"
server_key: "f690a34952386bf16d25b145cbf3cd10"
server_key_iv: "4d36fb9263f7c3e3d48a57ee33948462"
server_key_version: "encryptionkey@0"
I20260926 16:45:11.313170 4852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance:
uuid: "ed20749a2d644876849e5f81fe7bd137"
format_stamp: "Formatted at 2026-09-26 16:45:11 on dist-test-slave-n1hw"
server_key: "f690a34952386bf16d25b145cbf3cd10"
server_key_iv: "4d36fb9263f7c3e3d48a57ee33948462"
server_key_version: "encryptionkey@0"
I20260926 16:45:11.317008 4852 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 16:45:11.319473 4867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:45:11.320761 4852 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:45:11.320854 4852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-2/wal
uuid: "ed20749a2d644876849e5f81fe7bd137"
format_stamp: "Formatted at 2026-09-26 16:45:11 on dist-test-slave-n1hw"
server_key: "f690a34952386bf16d25b145cbf3cd10"
server_key_iv: "4d36fb9263f7c3e3d48a57ee33948462"
server_key_version: "encryptionkey@0"
I20260926 16:45:11.320938 4852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:45:11.349661 4852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:45:11.352943 4852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:45:11.353161 4852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:45:11.353822 4852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:45:11.354655 4852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:45:11.354727 4852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:45:11.354801 4852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:45:11.354851 4852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:45:11.363665 4852 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.67:39145
I20260926 16:45:11.363686 4981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.67:39145 every 8 connection(s)
I20260926 16:45:11.364557 4852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
I20260926 16:45:11.371122 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 4852
I20260926 16:45:11.371214 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance
I20260926 16:45:11.371495 21553 external_mini_cluster.cc:1428] Setting key dcba8963781241db470f9b6fe1d9e73a
Sep 26 16:45:11 dist-test-slave-n1hw krb5kdc[4473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441111, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.67@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:45:11.377795 4557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:11.366336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:55749 (local address 127.21.12.126:35725)
0926 16:45:11.366502 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 16:45:11.366506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:45:11.367312 (+ 806us) server_negotiation.cc:407] Connection header received
0926 16:45:11.368583 (+ 1271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:45:11.368586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:45:11.368632 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:45:11.368705 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:45:11.369941 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:11.370512 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:11.371784 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:11.372060 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:11.372165 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:11.374826 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:45:11.374829 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:45:11.374831 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:45:11.374855 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:45:11.376054 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:45:11.376499 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:45:11.376501 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:45:11.376502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:45:11.376576 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:45:11.376853 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:45:11.376856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:45:11.376857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:45:11.377018 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:45:11.377094 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 16:45:11.377512 (+ 418us) server_negotiation.cc:299] Negotiation successful
0926 16:45:11.377627 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 16:45:11.378582 4984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:11.366730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.67:55749)
0926 16:45:11.367159 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:11.367203 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 16:45:11.368394 (+ 1191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:45:11.368901 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:45:11.368907 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:45:11.369249 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:45:11.369720 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:11.369734 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:11.370620 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:11.370623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:11.371641 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:11.371651 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:11.372229 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:11.372233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:11.372458 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:11.373145 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:45:11.373161 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:45:11.374678 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:45:11.376153 (+ 1475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:45:11.376158 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:45:11.376169 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:45:11.376416 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:45:11.376652 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:45:11.376654 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:45:11.376655 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:45:11.376777 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:45:11.377101 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:45:11.377105 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:45:11.377321 (+ 216us) client_negotiation.cc:769] Sending connection context
0926 16:45:11.377468 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 16:45:11.377762 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":106,"threads_started":1}
I20260926 16:45:11.379673 4982 heartbeater.cc:347] Connected to a master server at 127.21.12.126:35725
I20260926 16:45:11.379884 4982 heartbeater.cc:464] Registering TS with master...
I20260926 16:45:11.380295 4982 heartbeater.cc:511] Master 127.21.12.126:35725 requested a full tablet report, sending...
I20260926 16:45:11.381224 4518 ts_manager.cc:194] Registered new tserver with Master: ed20749a2d644876849e5f81fe7bd137 (127.21.12.67:39145)
I20260926 16:45:11.381744 4518 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.67@KRBTEST.COM'} at 127.21.12.67:55749
I20260926 16:45:11.385192 21553 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:45:11.394256 4557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:11.387134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37152 (local address 127.21.12.126:35725)
0926 16:45:11.387305 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 16:45:11.387310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:45:11.387446 (+ 136us) server_negotiation.cc:407] Connection header received
0926 16:45:11.387591 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:45:11.387595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:45:11.387638 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:45:11.387732 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:45:11.388396 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:11.388901 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:11.389726 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:11.389993 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:11.390103 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:11.391369 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:45:11.391373 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:45:11.391375 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:45:11.391401 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:45:11.392530 (+ 1129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:45:11.392951 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:45:11.392955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:45:11.392957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:45:11.393041 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:45:11.393312 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:45:11.393317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:45:11.393319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:45:11.393533 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:45:11.393675 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 16:45:11.393961 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 16:45:11.394088 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 16:45:11.396961 4518 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37152:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:45:11.400059 4518 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:45:11.414702 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:11.408687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.67:39145 (local address 127.0.0.1:36718)
0926 16:45:11.409662 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:11.409711 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 16:45:11.409872 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:45:11.410679 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:45:11.410686 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:45:11.410711 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:45:11.410887 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:11.410895 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:11.411917 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:11.411923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:11.413028 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:11.413037 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:11.414102 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:11.414106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:11.414267 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:11.414307 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 16:45:11.414430 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 16:45:11.414553 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"mutex_wait_us":24,"thread_start_us":76,"threads_started":1}
I20260926 16:45:11.414712 4996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:11.409448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36718 (local address 127.21.12.67:39145)
0926 16:45:11.409821 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0926 16:45:11.409834 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:45:11.409862 (+ 28us) server_negotiation.cc:407] Connection header received
0926 16:45:11.410099 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:45:11.410105 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:45:11.410333 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:45:11.410486 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:45:11.411016 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:11.411790 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:11.413155 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:11.413949 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:11.414068 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:11.414236 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 16:45:11.414462 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 16:45:11.414555 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":136,"threads_started":1}
I20260926 16:45:11.415340 4997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:11.408867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:35041 (local address 127.0.0.1:55232)
0926 16:45:11.410220 (+ 1353us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:11.410235 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:45:11.410341 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:45:11.410860 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:45:11.410864 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:45:11.410894 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:45:11.410990 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:11.410998 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:11.412319 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:11.412322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:11.413615 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:11.413626 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:11.414885 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:11.414889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:11.414979 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:11.415002 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 16:45:11.415083 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 16:45:11.415152 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1270,"mutex_wait_us":71,"spinlock_wait_cycles":8704,"thread_start_us":92,"threads_started":1}
I20260926 16:45:11.415799 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:11.409983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:34405 (local address 127.0.0.1:45538)
0926 16:45:11.410788 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:11.410804 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:45:11.410902 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:45:11.411549 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:45:11.411552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:45:11.411569 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:45:11.411650 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:11.411656 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:11.413066 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:11.413070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:11.414236 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:11.414247 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:11.415456 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:11.415460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:11.415553 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:11.415580 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 16:45:11.415651 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 16:45:11.415702 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"mutex_wait_us":23,"spinlock_wait_cycles":10880,"thread_start_us":99,"threads_started":1}
I20260926 16:45:11.416167 4998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:11.410145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45538 (local address 127.21.12.65:34405)
0926 16:45:11.410706 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0926 16:45:11.410719 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:45:11.410995 (+ 276us) server_negotiation.cc:407] Connection header received
0926 16:45:11.411075 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:45:11.411080 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:45:11.411257 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:45:11.411401 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:45:11.411789 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:11.412924 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:11.414370 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:11.415157 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:11.415259 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:11.415389 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 16:45:11.415682 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 16:45:11.415779 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":118,"threads_started":1}
I20260926 16:45:11.417974 4995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:11.408890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55232 (local address 127.21.12.66:35041)
0926 16:45:11.409415 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0926 16:45:11.409427 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:45:11.410436 (+ 1009us) server_negotiation.cc:407] Connection header received
0926 16:45:11.410518 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:45:11.410525 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:45:11.410718 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:45:11.410885 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:45:11.411077 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:11.412153 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:11.413743 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:11.414604 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:11.414699 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:11.414825 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 16:45:11.415138 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 16:45:11.417848 (+ 2710us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":150,"threads_started":1}
I20260926 16:45:11.418286 4915 tablet_service.cc:1467] Processing CreateTablet for tablet 591af678b6d74272a61f9db3d2ed9db3 (DEFAULT_TABLE table=test-table [id=84d3f148ccaa4833952ef0c257e4948a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:45:11.418964 4645 tablet_service.cc:1467] Processing CreateTablet for tablet 591af678b6d74272a61f9db3d2ed9db3 (DEFAULT_TABLE table=test-table [id=84d3f148ccaa4833952ef0c257e4948a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:45:11.419430 4915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 591af678b6d74272a61f9db3d2ed9db3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:45:11.419986 4645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 591af678b6d74272a61f9db3d2ed9db3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:45:11.420302 4780 tablet_service.cc:1467] Processing CreateTablet for tablet 591af678b6d74272a61f9db3d2ed9db3 (DEFAULT_TABLE table=test-table [id=84d3f148ccaa4833952ef0c257e4948a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:45:11.421124 4780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 591af678b6d74272a61f9db3d2ed9db3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:45:11.427964 5001 tablet_bootstrap.cc:492] T 591af678b6d74272a61f9db3d2ed9db3 P 763e6b4243744af68be266b9d09c1aa7: Bootstrap starting.
I20260926 16:45:11.428143 4999 tablet_bootstrap.cc:492] T 591af678b6d74272a61f9db3d2ed9db3 P 9847782936414dd38e658a1faf3d2234: Bootstrap starting.
I20260926 16:45:11.429891 4999 tablet_bootstrap.cc:654] T 591af678b6d74272a61f9db3d2ed9db3 P 9847782936414dd38e658a1faf3d2234: Neither blocks nor log segments found. Creating new log.
I20260926 16:45:11.430064 5000 tablet_bootstrap.cc:492] T 591af678b6d74272a61f9db3d2ed9db3 P ed20749a2d644876849e5f81fe7bd137: Bootstrap starting.
I20260926 16:45:11.430349 5001 tablet_bootstrap.cc:654] T 591af678b6d74272a61f9db3d2ed9db3 P 763e6b4243744af68be266b9d09c1aa7: Neither blocks nor log segments found. Creating new log.
I20260926 16:45:11.430776 4999 log.cc:824] T 591af678b6d74272a61f9db3d2ed9db3 P 9847782936414dd38e658a1faf3d2234: Log is configured to *not* fsync() on all Append() calls
I20260926 16:45:11.431061 5001 log.cc:824] T 591af678b6d74272a61f9db3d2ed9db3 P 763e6b4243744af68be266b9d09c1aa7: Log is configured to *not* fsync() on all Append() calls
I20260926 16:45:11.431891 5000 tablet_bootstrap.cc:654] T 591af678b6d74272a61f9db3d2ed9db3 P ed20749a2d644876849e5f81fe7bd137: Neither blocks nor log segments found. Creating new log.
I20260926 16:45:11.433064 5000 log.cc:824] T 591af678b6d74272a61f9db3d2ed9db3 P ed20749a2d644876849e5f81fe7bd137: Log is configured to *not* fsync() on all Append() calls
I20260926 16:45:11.433130 5001 tablet_bootstrap.cc:492] T 591af678b6d74272a61f9db3d2ed9db3 P 763e6b4243744af68be266b9d09c1aa7: No bootstrap required, opened a new log
I20260926 16:45:11.433315 5001 ts_tablet_manager.cc:1397] T 591af678b6d74272a61f9db3d2ed9db3 P 763e6b4243744af68be266b9d09c1aa7: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260926 16:45:11.434757 4999 tablet_bootstrap.cc:492] T 591af678b6d74272a61f9db3d2ed9db3 P 9847782936414dd38e658a1faf3d2234: No bootstrap required, opened a new log
I20260926 16:45:11.434901 5000 tablet_bootstrap.cc:492] T 591af678b6d74272a61f9db3d2ed9db3 P ed20749a2d644876849e5f81fe7bd137: No bootstrap required, opened a new log
I20260926 16:45:11.434973 4999 ts_tablet_manager.cc:1397] T 591af678b6d74272a61f9db3d2ed9db3 P 9847782936414dd38e658a1faf3d2234: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20260926 16:45:11.435074 5000 ts_tablet_manager.cc:1397] T 591af678b6d74272a61f9db3d2ed9db3 P ed20749a2d644876849e5f81fe7bd137: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 16:45:11.437108 5001 raft_consensus.cc:348] T 591af678b6d74272a61f9db3d2ed9db3 P 763e6b4243744af68be266b9d09c1aa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20749a2d644876849e5f81fe7bd137" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 39145 } } peers { permanent_uuid: "763e6b4243744af68be266b9d09c1aa7" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 35041 } } peers { permanent_uuid: "9847782936414dd38e658a1faf3d2234" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 34405 } }
I20260926 16:45:11.437390 5001 raft_consensus.cc:374] T 591af678b6d74272a61f9db3d2ed9db3 P 763e6b4243744af68be266b9d09c1aa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:45:11.437479 5001 raft_consensus.cc:729] T 591af678b6d74272a61f9db3d2ed9db3 P 763e6b4243744af68be266b9d09c1aa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 763e6b4243744af68be266b9d09c1aa7, State: Initialized, Role: FOLLOWER
I20260926 16:45:11.438045 5001 consensus_queue.cc:260] T 591af678b6d74272a61f9db3d2ed9db3 P 763e6b4243744af68be266b9d09c1aa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20749a2d644876849e5f81fe7bd137" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 39145 } } peers { permanent_uuid: "763e6b4243744af68be266b9d09c1aa7" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 35041 } } peers { permanent_uuid: "9847782936414dd38e658a1faf3d2234" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 34405 } }
I20260926 16:45:11.438283 4999 raft_consensus.cc:348] T 591af678b6d74272a61f9db3d2ed9db3 P 9847782936414dd38e658a1faf3d2234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20749a2d644876849e5f81fe7bd137" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 39145 } } peers { permanent_uuid: "763e6b4243744af68be266b9d09c1aa7" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 35041 } } peers { permanent_uuid: "9847782936414dd38e658a1faf3d2234" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 34405 } }
I20260926 16:45:11.438580 4999 raft_consensus.cc:374] T 591af678b6d74272a61f9db3d2ed9db3 P 9847782936414dd38e658a1faf3d2234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:45:11.438681 4999 raft_consensus.cc:729] T 591af678b6d74272a61f9db3d2ed9db3 P 9847782936414dd38e658a1faf3d2234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9847782936414dd38e658a1faf3d2234, State: Initialized, Role: FOLLOWER
I20260926 16:45:11.438748 4847 heartbeater.cc:503] Master 127.21.12.126:35725 was elected leader, sending a full tablet report...
I20260926 16:45:11.439055 5001 ts_tablet_manager.cc:1428] T 591af678b6d74272a61f9db3d2ed9db3 P 763e6b4243744af68be266b9d09c1aa7: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 16:45:11.438869 5000 raft_consensus.cc:348] T 591af678b6d74272a61f9db3d2ed9db3 P ed20749a2d644876849e5f81fe7bd137 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20749a2d644876849e5f81fe7bd137" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 39145 } } peers { permanent_uuid: "763e6b4243744af68be266b9d09c1aa7" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 35041 } } peers { permanent_uuid: "9847782936414dd38e658a1faf3d2234" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 34405 } }
I20260926 16:45:11.439114 5000 raft_consensus.cc:374] T 591af678b6d74272a61f9db3d2ed9db3 P ed20749a2d644876849e5f81fe7bd137 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:45:11.439164 5000 raft_consensus.cc:729] T 591af678b6d74272a61f9db3d2ed9db3 P ed20749a2d644876849e5f81fe7bd137 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed20749a2d644876849e5f81fe7bd137, State: Initialized, Role: FOLLOWER
I20260926 16:45:11.439167 4999 consensus_queue.cc:260] T 591af678b6d74272a61f9db3d2ed9db3 P 9847782936414dd38e658a1faf3d2234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20749a2d644876849e5f81fe7bd137" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 39145 } } peers { permanent_uuid: "763e6b4243744af68be266b9d09c1aa7" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 35041 } } peers { permanent_uuid: "9847782936414dd38e658a1faf3d2234" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 34405 } }
I20260926 16:45:11.439551 5000 consensus_queue.cc:260] T 591af678b6d74272a61f9db3d2ed9db3 P ed20749a2d644876849e5f81fe7bd137 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20749a2d644876849e5f81fe7bd137" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 39145 } } peers { permanent_uuid: "763e6b4243744af68be266b9d09c1aa7" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 35041 } } peers { permanent_uuid: "9847782936414dd38e658a1faf3d2234" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 34405 } }
I20260926 16:45:11.439827 4712 heartbeater.cc:503] Master 127.21.12.126:35725 was elected leader, sending a full tablet report...
I20260926 16:45:11.440119 4999 ts_tablet_manager.cc:1428] T 591af678b6d74272a61f9db3d2ed9db3 P 9847782936414dd38e658a1faf3d2234: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260926 16:45:11.440537 5000 ts_tablet_manager.cc:1428] T 591af678b6d74272a61f9db3d2ed9db3 P ed20749a2d644876849e5f81fe7bd137: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 16:45:11.440830 4982 heartbeater.cc:503] Master 127.21.12.126:35725 was elected leader, sending a full tablet report...
I20260926 16:45:11.528481 5006 raft_consensus.cc:482] T 591af678b6d74272a61f9db3d2ed9db3 P 9847782936414dd38e658a1faf3d2234 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:45:11.528712 5006 raft_consensus.cc:504] T 591af678b6d74272a61f9db3d2ed9db3 P 9847782936414dd38e658a1faf3d2234 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20749a2d644876849e5f81fe7bd137" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 39145 } } peers { permanent_uuid: "763e6b4243744af68be266b9d09c1aa7" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 35041 } } peers { permanent_uuid: "9847782936414dd38e658a1faf3d2234" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 34405 } }
I20260926 16:45:11.529942 5006 leader_election.cc:288] T 591af678b6d74272a61f9db3d2ed9db3 P 9847782936414dd38e658a1faf3d2234 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed20749a2d644876849e5f81fe7bd137 (127.21.12.67:39145), 763e6b4243744af68be266b9d09c1aa7 (127.21.12.66:35041)
I20260926 16:45:11.534175 4995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:11.530333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:35489 (local address 127.21.12.66:35041)
0926 16:45:11.530531 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 16:45:11.530534 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:45:11.530625 (+ 91us) server_negotiation.cc:407] Connection header received
0926 16:45:11.530847 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:45:11.530871 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:45:11.530936 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:45:11.531013 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:45:11.531364 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:11.532049 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:11.533126 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:11.533527 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:11.533659 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:11.533723 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 16:45:11.533963 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 16:45:11.534044 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260926 16:45:11.534175 4996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:11.530286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:57399 (local address 127.21.12.67:39145)
0926 16:45:11.530422 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 16:45:11.530427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:45:11.530673 (+ 246us) server_negotiation.cc:407] Connection header received
0926 16:45:11.530795 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:45:11.530798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:45:11.530859 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:45:11.530970 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:45:11.531342 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:11.531985 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:11.533130 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:11.533528 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:11.533656 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:11.533708 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 16:45:11.533955 (+ 247us) server_negotiation.cc:299] Negotiation successful
0926 16:45:11.534040 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 16:45:11.534235 5009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:11.530179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.67:39145 (local address 127.21.12.65:57399)
0926 16:45:11.530575 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:11.530588 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 16:45:11.530675 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:45:11.530985 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:45:11.530988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:45:11.531006 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:45:11.531167 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:11.531176 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:11.532096 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:11.532102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:11.533007 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:11.533017 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:11.533656 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:11.533659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:11.533838 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:11.533855 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 16:45:11.533939 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 16:45:11.534060 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":99,"spinlock_wait_cycles":1920,"thread_start_us":111,"threads_started":1}
I20260926 16:45:11.534246 5008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:11.530171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:35041 (local address 127.21.12.65:35489)
0926 16:45:11.530526 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:11.530542 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:45:11.530676 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:45:11.531098 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:45:11.531101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:45:11.531117 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:45:11.531262 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:11.531273 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:11.532162 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:11.532167 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:11.532996 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:11.533005 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:11.533659 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:11.533663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:11.533816 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:11.533843 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 16:45:11.533936 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 16:45:11.534072 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":86,"spinlock_wait_cycles":1920,"thread_start_us":142,"threads_started":1}
I20260926 16:45:11.535024 4935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "591af678b6d74272a61f9db3d2ed9db3" candidate_uuid: "9847782936414dd38e658a1faf3d2234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed20749a2d644876849e5f81fe7bd137" is_pre_election: true
I20260926 16:45:11.535024 4800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "591af678b6d74272a61f9db3d2ed9db3" candidate_uuid: "9847782936414dd38e658a1faf3d2234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "763e6b4243744af68be266b9d09c1aa7" is_pre_election: true
I20260926 16:45:11.535337 4935 raft_consensus.cc:2454] T 591af678b6d74272a61f9db3d2ed9db3 P ed20749a2d644876849e5f81fe7bd137 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9847782936414dd38e658a1faf3d2234 in term 0.
I20260926 16:45:11.535337 4800 raft_consensus.cc:2454] T 591af678b6d74272a61f9db3d2ed9db3 P 763e6b4243744af68be266b9d09c1aa7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9847782936414dd38e658a1faf3d2234 in term 0.
I20260926 16:45:11.535887 4601 leader_election.cc:302] T 591af678b6d74272a61f9db3d2ed9db3 P 9847782936414dd38e658a1faf3d2234 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 763e6b4243744af68be266b9d09c1aa7, 9847782936414dd38e658a1faf3d2234, ed20749a2d644876849e5f81fe7bd137; no voters:
I20260926 16:45:11.536166 5006 raft_consensus.cc:2781] T 591af678b6d74272a61f9db3d2ed9db3 P 9847782936414dd38e658a1faf3d2234 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:45:11.536249 5006 raft_consensus.cc:482] T 591af678b6d74272a61f9db3d2ed9db3 P 9847782936414dd38e658a1faf3d2234 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:45:11.536280 5006 raft_consensus.cc:3037] T 591af678b6d74272a61f9db3d2ed9db3 P 9847782936414dd38e658a1faf3d2234 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:45:11.537335 5006 raft_consensus.cc:504] T 591af678b6d74272a61f9db3d2ed9db3 P 9847782936414dd38e658a1faf3d2234 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20749a2d644876849e5f81fe7bd137" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 39145 } } peers { permanent_uuid: "763e6b4243744af68be266b9d09c1aa7" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 35041 } } peers { permanent_uuid: "9847782936414dd38e658a1faf3d2234" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 34405 } }
I20260926 16:45:11.537679 5006 leader_election.cc:288] T 591af678b6d74272a61f9db3d2ed9db3 P 9847782936414dd38e658a1faf3d2234 [CANDIDATE]: Term 1 election: Requested vote from peers ed20749a2d644876849e5f81fe7bd137 (127.21.12.67:39145), 763e6b4243744af68be266b9d09c1aa7 (127.21.12.66:35041)
I20260926 16:45:11.538087 4935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "591af678b6d74272a61f9db3d2ed9db3" candidate_uuid: "9847782936414dd38e658a1faf3d2234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed20749a2d644876849e5f81fe7bd137"
I20260926 16:45:11.538087 4800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "591af678b6d74272a61f9db3d2ed9db3" candidate_uuid: "9847782936414dd38e658a1faf3d2234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "763e6b4243744af68be266b9d09c1aa7"
I20260926 16:45:11.538237 4800 raft_consensus.cc:3037] T 591af678b6d74272a61f9db3d2ed9db3 P 763e6b4243744af68be266b9d09c1aa7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:45:11.538236 4935 raft_consensus.cc:3037] T 591af678b6d74272a61f9db3d2ed9db3 P ed20749a2d644876849e5f81fe7bd137 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:45:11.539635 4800 raft_consensus.cc:2454] T 591af678b6d74272a61f9db3d2ed9db3 P 763e6b4243744af68be266b9d09c1aa7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9847782936414dd38e658a1faf3d2234 in term 1.
I20260926 16:45:11.539628 4935 raft_consensus.cc:2454] T 591af678b6d74272a61f9db3d2ed9db3 P ed20749a2d644876849e5f81fe7bd137 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9847782936414dd38e658a1faf3d2234 in term 1.
I20260926 16:45:11.539961 4600 leader_election.cc:302] T 591af678b6d74272a61f9db3d2ed9db3 P 9847782936414dd38e658a1faf3d2234 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 763e6b4243744af68be266b9d09c1aa7, 9847782936414dd38e658a1faf3d2234; no voters:
I20260926 16:45:11.540246 5006 raft_consensus.cc:2781] T 591af678b6d74272a61f9db3d2ed9db3 P 9847782936414dd38e658a1faf3d2234 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:45:11.540462 5006 raft_consensus.cc:686] T 591af678b6d74272a61f9db3d2ed9db3 P 9847782936414dd38e658a1faf3d2234 [term 1 LEADER]: Becoming Leader. State: Replica: 9847782936414dd38e658a1faf3d2234, State: Running, Role: LEADER
I20260926 16:45:11.540833 5006 consensus_queue.cc:237] T 591af678b6d74272a61f9db3d2ed9db3 P 9847782936414dd38e658a1faf3d2234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20749a2d644876849e5f81fe7bd137" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 39145 } } peers { permanent_uuid: "763e6b4243744af68be266b9d09c1aa7" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 35041 } } peers { permanent_uuid: "9847782936414dd38e658a1faf3d2234" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 34405 } }
I20260926 16:45:11.544335 4518 catalog_manager.cc:5495] T 591af678b6d74272a61f9db3d2ed9db3 P 9847782936414dd38e658a1faf3d2234 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9847782936414dd38e658a1faf3d2234 (127.21.12.65). New cstate: current_term: 1 leader_uuid: "9847782936414dd38e658a1faf3d2234" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20749a2d644876849e5f81fe7bd137" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 39145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "763e6b4243744af68be266b9d09c1aa7" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 35041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9847782936414dd38e658a1faf3d2234" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 34405 } health_report { overall_health: HEALTHY } } }
I20260926 16:45:11.569651 4998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:11.565965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45552 (local address 127.21.12.65:34405)
0926 16:45:11.566117 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 16:45:11.566122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:45:11.566137 (+ 15us) server_negotiation.cc:407] Connection header received
0926 16:45:11.566358 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:45:11.566360 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:45:11.566422 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:45:11.566539 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:45:11.566798 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:11.567302 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:11.568166 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:11.568389 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:11.568501 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:11.568824 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:45:11.569245 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:45:11.569386 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 16:45:11.569429 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 16:45:11.569487 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 16:45:11.575471 4935 raft_consensus.cc:1261] T 591af678b6d74272a61f9db3d2ed9db3 P ed20749a2d644876849e5f81fe7bd137 [term 1 FOLLOWER]: Refusing update from remote peer 9847782936414dd38e658a1faf3d2234: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:45:11.575470 4800 raft_consensus.cc:1261] T 591af678b6d74272a61f9db3d2ed9db3 P 763e6b4243744af68be266b9d09c1aa7 [term 1 FOLLOWER]: Refusing update from remote peer 9847782936414dd38e658a1faf3d2234: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:45:11.576195 5010 consensus_queue.cc:1035] T 591af678b6d74272a61f9db3d2ed9db3 P 9847782936414dd38e658a1faf3d2234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "763e6b4243744af68be266b9d09c1aa7" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 35041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:45:11.576380 5006 consensus_queue.cc:1035] T 591af678b6d74272a61f9db3d2ed9db3 P 9847782936414dd38e658a1faf3d2234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed20749a2d644876849e5f81fe7bd137" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 39145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:45:11.583411 5014 mvcc.cc:205] Tried to move back new op lower bound from 7333646793006120960 to 7333646792874463232. Current Snapshot: MvccSnapshot[applied={T|T < 7333646793006120960}]
I20260926 16:45:11.584132 5015 mvcc.cc:205] Tried to move back new op lower bound from 7333646793006120960 to 7333646792874463232. Current Snapshot: MvccSnapshot[applied={T|T < 7333646793006120960}]
I20260926 16:45:11.592078 4516 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:45:11.592262 4516 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:45:11.594702 4516 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4: Sending DeleteTablet for 3 replicas of tablet 591af678b6d74272a61f9db3d2ed9db3
I20260926 16:45:11.595446 4915 tablet_service.cc:1514] Processing DeleteTablet for tablet 591af678b6d74272a61f9db3d2ed9db3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:45:11 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:36718
I20260926 16:45:11.595446 4780 tablet_service.cc:1514] Processing DeleteTablet for tablet 591af678b6d74272a61f9db3d2ed9db3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:45:11 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:55232
I20260926 16:45:11.595611 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 4581
I20260926 16:45:11.597515 5023 tablet_replica.cc:330] T 591af678b6d74272a61f9db3d2ed9db3 P ed20749a2d644876849e5f81fe7bd137: stopping tablet replica
I20260926 16:45:11.597878 5023 raft_consensus.cc:2229] T 591af678b6d74272a61f9db3d2ed9db3 P ed20749a2d644876849e5f81fe7bd137 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:45:11.597939 5024 tablet_replica.cc:330] T 591af678b6d74272a61f9db3d2ed9db3 P 763e6b4243744af68be266b9d09c1aa7: stopping tablet replica
I20260926 16:45:11.598237 5024 raft_consensus.cc:2229] T 591af678b6d74272a61f9db3d2ed9db3 P 763e6b4243744af68be266b9d09c1aa7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:45:11.598295 5023 raft_consensus.cc:2258] T 591af678b6d74272a61f9db3d2ed9db3 P ed20749a2d644876849e5f81fe7bd137 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:45:11.598522 5024 raft_consensus.cc:2258] T 591af678b6d74272a61f9db3d2ed9db3 P 763e6b4243744af68be266b9d09c1aa7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:45:11.599896 5023 ts_tablet_manager.cc:1905] T 591af678b6d74272a61f9db3d2ed9db3 P ed20749a2d644876849e5f81fe7bd137: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:45:11.599896 5024 ts_tablet_manager.cc:1905] T 591af678b6d74272a61f9db3d2ed9db3 P 763e6b4243744af68be266b9d09c1aa7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 16:45:11.602147 4502 connection.cc:535] client connection to 127.21.12.65:34405 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 16:45:11.602299 4502 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 16:45:11.602586 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 4717
I20260926 16:45:11.602622 5024 ts_tablet_manager.cc:1918] T 591af678b6d74272a61f9db3d2ed9db3 P 763e6b4243744af68be266b9d09c1aa7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:45:11.602622 5023 ts_tablet_manager.cc:1918] T 591af678b6d74272a61f9db3d2ed9db3 P ed20749a2d644876849e5f81fe7bd137: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:45:11.602708 5024 log.cc:1196] T 591af678b6d74272a61f9db3d2ed9db3 P 763e6b4243744af68be266b9d09c1aa7: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-1/wal/wals/591af678b6d74272a61f9db3d2ed9db3
I20260926 16:45:11.602717 5023 log.cc:1196] T 591af678b6d74272a61f9db3d2ed9db3 P ed20749a2d644876849e5f81fe7bd137: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-2/wal/wals/591af678b6d74272a61f9db3d2ed9db3
I20260926 16:45:11.603029 5023 ts_tablet_manager.cc:1939] T 591af678b6d74272a61f9db3d2ed9db3 P ed20749a2d644876849e5f81fe7bd137: Deleting consensus metadata
I20260926 16:45:11.604281 4505 catalog_manager.cc:4841] TS ed20749a2d644876849e5f81fe7bd137 (127.21.12.67:39145): tablet 591af678b6d74272a61f9db3d2ed9db3 (table test-table [id=84d3f148ccaa4833952ef0c257e4948a]) successfully deleted
I20260926 16:45:11.606559 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:11.606278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:34405 (local address 127.0.0.1:45562)
0926 16:45:11.606426 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:11.606490 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.65:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260926 16:45:11.606837 4502 catalog_manager.cc:4584] TS 9847782936414dd38e658a1faf3d2234 (127.21.12.65:34405): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 591af678b6d74272a61f9db3d2ed9db3: Network error: Client connection negotiation failed: client connection to 127.21.12.65:34405: connect: Connection refused (error 111)
I20260926 16:45:11.609253 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 4852
I20260926 16:45:11.610882 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:11.609307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:35041 (local address 127.0.0.1:55234)
0926 16:45:11.610770 (+ 1463us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:11.610819 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.66:35041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1377}
I20260926 16:45:11.614881 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 4486
I20260926 16:45:31.621404 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:35725
--webserver_interface=127.21.12.126
--webserver_port=41287
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:45327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:35725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:45:31.728704 5026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:45:31.728960 5026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:45:31.729000 5026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:45:31.733038 5026 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 16:45:31.733083 5026 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 16:45:31.733098 5026 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 16:45:31.733112 5026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:45:31.733158 5026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:45:31.733194 5026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:45:31.733215 5026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:45:31.733242 5026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:45:31.741173 5026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:45327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:35725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:35725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.126
--webserver_port=41287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:45:31.742558 5026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:45:31.743582 5026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:45:31.756173 5032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:45:31.756254 5035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:45:31.756249 5033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:45:31.756372 5026 server_base.cc:1034] running on GCE node
I20260926 16:45:31.757162 5026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:45:31.758445 5026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:45:31.759615 5026 hybrid_clock.cc:648] HybridClock initialized: now 1790441131759596 us; error 30 us; skew 500 ppm
Sep 26 16:45:31 dist-test-slave-n1hw krb5kdc[4473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441131, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:45:31.762348 5026 init.cc:377] Logged in from keytab as kudu/127.21.12.126@KRBTEST.COM (short username kudu)
I20260926 16:45:31.763235 5026 webserver.cc:460] Webserver started at http://127.21.12.126:41287/ using document root <none> and password file <none>
I20260926 16:45:31.763756 5026 fs_manager.cc:359] Metadata directory not provided
I20260926 16:45:31.763824 5026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:45:31.767679 5026 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:45:31.769723 5042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:45:31.770993 5026 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:45:31.771095 5026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "25a40b6cfa6544cdb74879e3751924c4"
format_stamp: "Formatted at 2026-09-26 16:45:10 on dist-test-slave-n1hw"
server_key: "7a3a695de4ee7cd939009ecc2d9e3f4e"
server_key_iv: "c6ba6db48d53dbb31bc22985e3583c26"
server_key_version: "encryptionkey@0"
I20260926 16:45:31.771647 5026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:45:31.783151 5026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:45:31.786118 5026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:45:31.786270 5026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:45:31.793615 5026 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.126:35725
I20260926 16:45:31.793936 5094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.126:35725 every 8 connection(s)
I20260926 16:45:31.794907 5026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:45:31.797549 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 5026
I20260926 16:45:31.798034 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.65:34405
--local_ip_for_outbound_sockets=127.21.12.65
--tserver_master_addrs=127.21.12.126:35725
--webserver_port=32957
--webserver_interface=127.21.12.65
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab
--principal=kudu/127.21.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:45327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:45:31.799453 5095 sys_catalog.cc:263] Verifying existing consensus state
I20260926 16:45:31.801098 5095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4: Bootstrap starting.
I20260926 16:45:31.818110 5095 log.cc:824] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4: Log is configured to *not* fsync() on all Append() calls
I20260926 16:45:31.826534 5095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 16:45:31.826905 5095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4: Bootstrap complete.
I20260926 16:45:31.830909 5095 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25a40b6cfa6544cdb74879e3751924c4" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 35725 } }
I20260926 16:45:31.831315 5095 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25a40b6cfa6544cdb74879e3751924c4, State: Initialized, Role: FOLLOWER
I20260926 16:45:31.831830 5095 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25a40b6cfa6544cdb74879e3751924c4" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 35725 } }
I20260926 16:45:31.831992 5095 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:45:31.832085 5095 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:45:31.832201 5095 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [term 1 FOLLOWER]: Advancing to term 2
I20260926 16:45:31.834089 5095 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25a40b6cfa6544cdb74879e3751924c4" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 35725 } }
I20260926 16:45:31.834409 5095 leader_election.cc:302] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25a40b6cfa6544cdb74879e3751924c4; no voters:
I20260926 16:45:31.834733 5095 leader_election.cc:288] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 16:45:31.834805 5099 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 16:45:31.835031 5099 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [term 2 LEADER]: Becoming Leader. State: Replica: 25a40b6cfa6544cdb74879e3751924c4, State: Running, Role: LEADER
I20260926 16:45:31.835376 5099 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25a40b6cfa6544cdb74879e3751924c4" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 35725 } }
I20260926 16:45:31.835970 5095 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:45:31.836871 5100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25a40b6cfa6544cdb74879e3751924c4. Latest consensus state: current_term: 2 leader_uuid: "25a40b6cfa6544cdb74879e3751924c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25a40b6cfa6544cdb74879e3751924c4" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 35725 } } }
I20260926 16:45:31.836877 5101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "25a40b6cfa6544cdb74879e3751924c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25a40b6cfa6544cdb74879e3751924c4" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 35725 } } }
I20260926 16:45:31.837117 5101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [sys.catalog]: This master's current role is: LEADER
I20260926 16:45:31.837062 5100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [sys.catalog]: This master's current role is: LEADER
I20260926 16:45:31.837487 5108 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:45:31.842309 5108 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:45:31.843001 5108 catalog_manager.cc:1259] Loaded cluster ID: 8099a53dc86f448086e3dea5e9ed0398
I20260926 16:45:31.843060 5108 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:45:31.845463 5108 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:45:31.846189 5108 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4: Loaded TSK: 0
I20260926 16:45:31.860906 5108 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4: Generated new TSK 1
I20260926 16:45:31.861166 5108 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 16:45:31.910778 5097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:45:31.911082 5097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:45:31.911159 5097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:45:31.915270 5097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:45:31.915351 5097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:45:31.915446 5097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.65
I20260926 16:45:31.920593 5097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:45327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab
--principal=kudu/127.21.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.65:34405
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.65
--webserver_port=32957
--tserver_master_addrs=127.21.12.126:35725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.65
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:45:31.921653 5097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:45:31.922748 5097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:45:31.933272 5122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:45:31.933301 5123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:45:31.933395 5097 server_base.cc:1034] running on GCE node
W20260926 16:45:31.933696 5125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:45:31.934077 5097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:45:31.934717 5097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:45:31.935899 5097 hybrid_clock.cc:648] HybridClock initialized: now 1790441131935874 us; error 35 us; skew 500 ppm
Sep 26 16:45:31 dist-test-slave-n1hw krb5kdc[4473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441131, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:45:31.938810 5097 init.cc:377] Logged in from keytab as kudu/127.21.12.65@KRBTEST.COM (short username kudu)
I20260926 16:45:31.939738 5097 webserver.cc:460] Webserver started at http://127.21.12.65:32957/ using document root <none> and password file <none>
I20260926 16:45:31.940280 5097 fs_manager.cc:359] Metadata directory not provided
I20260926 16:45:31.940353 5097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:45:31.944316 5097 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 16:45:31.946475 5132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:45:31.947728 5097 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 16:45:31.947817 5097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-0/wal
uuid: "9847782936414dd38e658a1faf3d2234"
format_stamp: "Formatted at 2026-09-26 16:45:10 on dist-test-slave-n1hw"
server_key: "f95bbc156153ccc503343b53f20644e4"
server_key_iv: "2ed0b0449f353e1bf084a77ddb634014"
server_key_version: "encryptionkey@0"
I20260926 16:45:31.948375 5097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:45:31.960620 5097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:45:31.963856 5097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:45:31.964100 5097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:45:31.964745 5097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:45:31.965981 5139 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 16:45:31.971055 5097 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 16:45:31.971201 5097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.001s
I20260926 16:45:31.971287 5097 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 16:45:31.973042 5097 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 16:45:31.973114 5097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260926 16:45:31.973399 5139 tablet_bootstrap.cc:492] T 591af678b6d74272a61f9db3d2ed9db3 P 9847782936414dd38e658a1faf3d2234: Bootstrap starting.
I20260926 16:45:31.985054 5097 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.65:34405
I20260926 16:45:31.985229 5248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.65:34405 every 8 connection(s)
I20260926 16:45:31.986498 5097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
I20260926 16:45:31.994666 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 5097
I20260926 16:45:31.995060 5139 log.cc:824] T 591af678b6d74272a61f9db3d2ed9db3 P 9847782936414dd38e658a1faf3d2234: Log is configured to *not* fsync() on all Append() calls
I20260926 16:45:31.995234 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.66:35041
--local_ip_for_outbound_sockets=127.21.12.66
--tserver_master_addrs=127.21.12.126:35725
--webserver_port=44039
--webserver_interface=127.21.12.66
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab
--principal=kudu/127.21.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:45327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 16:45:31 dist-test-slave-n1hw krb5kdc[4473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441131, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:45:32.003325 5139 tablet_bootstrap.cc:492] T 591af678b6d74272a61f9db3d2ed9db3 P 9847782936414dd38e658a1faf3d2234: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 16:45:32.003803 5139 tablet_bootstrap.cc:492] T 591af678b6d74272a61f9db3d2ed9db3 P 9847782936414dd38e658a1faf3d2234: Bootstrap complete.
I20260926 16:45:32.005044 5139 ts_tablet_manager.cc:1397] T 591af678b6d74272a61f9db3d2ed9db3 P 9847782936414dd38e658a1faf3d2234: Time spent bootstrapping tablet: real 0.032s user 0.020s sys 0.008s
I20260926 16:45:32.006636 5252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:31.988970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.65:37241)
0926 16:45:31.989564 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:31.989617 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 16:45:31.990769 (+ 1152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:45:31.992263 (+ 1494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:45:31.992271 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:45:31.992699 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:45:31.993322 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:31.993341 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:31.995279 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:31.995283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:31.996802 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:31.996829 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:31.997596 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:31.997601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:31.997831 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:31.998472 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:45:31.998497 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:45:32.000455 (+ 1958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:45:32.003046 (+ 2591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:45:32.003054 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:45:32.003071 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:45:32.003485 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:45:32.003930 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:45:32.003933 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:45:32.003935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:45:32.004103 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:45:32.004881 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:45:32.004890 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:45:32.005175 (+ 285us) client_negotiation.cc:769] Sending connection context
0926 16:45:32.005374 (+ 199us) client_negotiation.cc:240] Negotiation successful
0926 16:45:32.005686 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":129,"threads_started":1}
I20260926 16:45:32.006947 5251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:31.988857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:37241 (local address 127.21.12.126:35725)
0926 16:45:31.989278 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0926 16:45:31.989300 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 16:45:31.989726 (+ 426us) server_negotiation.cc:407] Connection header received
0926 16:45:31.991324 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:45:31.991348 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:45:31.991786 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:45:31.992167 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:45:31.993521 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:31.995067 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:31.996971 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:31.997446 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:31.997626 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:32.000690 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:45:32.000697 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:45:32.000713 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:45:32.000746 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:45:32.002911 (+ 2165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:45:32.003625 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:45:32.003634 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:45:32.003641 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:45:32.003791 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:45:32.004218 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:45:32.004224 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:45:32.004228 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:45:32.004747 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:45:32.004917 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 16:45:32.005543 (+ 626us) server_negotiation.cc:299] Negotiation successful
0926 16:45:32.005959 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":97,"threads_started":1}
I20260926 16:45:32.008034 5139 raft_consensus.cc:348] T 591af678b6d74272a61f9db3d2ed9db3 P 9847782936414dd38e658a1faf3d2234 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20749a2d644876849e5f81fe7bd137" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 39145 } } peers { permanent_uuid: "763e6b4243744af68be266b9d09c1aa7" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 35041 } } peers { permanent_uuid: "9847782936414dd38e658a1faf3d2234" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 34405 } }
I20260926 16:45:32.008556 5139 raft_consensus.cc:729] T 591af678b6d74272a61f9db3d2ed9db3 P 9847782936414dd38e658a1faf3d2234 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9847782936414dd38e658a1faf3d2234, State: Initialized, Role: FOLLOWER
I20260926 16:45:32.009003 5139 consensus_queue.cc:260] T 591af678b6d74272a61f9db3d2ed9db3 P 9847782936414dd38e658a1faf3d2234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20749a2d644876849e5f81fe7bd137" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 39145 } } peers { permanent_uuid: "763e6b4243744af68be266b9d09c1aa7" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 35041 } } peers { permanent_uuid: "9847782936414dd38e658a1faf3d2234" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 34405 } }
I20260926 16:45:32.009891 5249 heartbeater.cc:347] Connected to a master server at 127.21.12.126:35725
I20260926 16:45:32.009955 5139 ts_tablet_manager.cc:1428] T 591af678b6d74272a61f9db3d2ed9db3 P 9847782936414dd38e658a1faf3d2234: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20260926 16:45:32.010142 5249 heartbeater.cc:464] Registering TS with master...
I20260926 16:45:32.010665 5249 heartbeater.cc:511] Master 127.21.12.126:35725 requested a full tablet report, sending...
I20260926 16:45:32.013157 5059 ts_manager.cc:194] Registered new tserver with Master: 9847782936414dd38e658a1faf3d2234 (127.21.12.65:34405)
I20260926 16:45:32.016522 5059 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.65@KRBTEST.COM'} at 127.21.12.65:37241
I20260926 16:45:32.019268 5249 heartbeater.cc:503] Master 127.21.12.126:35725 was elected leader, sending a full tablet report...
Sep 26 16:45:32 dist-test-slave-n1hw krb5kdc[4473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441131, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for kudu/127.21.12.65@KRBTEST.COM
I20260926 16:45:32.026640 5256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:32.015841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:34405 (local address 127.0.0.1:42248)
0926 16:45:32.016402 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:32.016448 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 16:45:32.016565 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:45:32.017079 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:45:32.017084 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:45:32.017504 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:45:32.017625 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:32.017632 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:32.018921 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:32.018924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:32.019855 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:32.019866 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:32.020628 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:32.020631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:32.020744 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:32.021270 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:45:32.021285 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:45:32.022807 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:45:32.024909 (+ 2102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:45:32.024917 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:45:32.024921 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:45:32.025135 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:45:32.025457 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:45:32.025460 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:45:32.025461 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:45:32.025547 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:45:32.026109 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:45:32.026114 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:45:32.026213 (+ 99us) client_negotiation.cc:769] Sending connection context
0926 16:45:32.026348 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 16:45:32.026513 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":114,"threads_started":1}
I20260926 16:45:32.026773 5257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:32.015940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42248 (local address 127.21.12.65:34405)
0926 16:45:32.016543 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0926 16:45:32.016556 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:45:32.016579 (+ 23us) server_negotiation.cc:407] Connection header received
0926 16:45:32.016714 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:45:32.016719 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:45:32.016917 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:45:32.017060 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:45:32.017769 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:32.018789 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:32.019976 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:32.020503 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:32.020614 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:32.022940 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:45:32.022950 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:45:32.022962 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:45:32.022986 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:45:32.024799 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:45:32.025239 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:45:32.025245 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:45:32.025247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:45:32.025338 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:45:32.025643 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:45:32.025647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:45:32.025649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:45:32.025983 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:45:32.026112 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 16:45:32.026453 (+ 341us) server_negotiation.cc:299] Negotiation successful
0926 16:45:32.026614 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":99,"threads_started":1}
I20260926 16:45:32.027699 5176 tablet_service.cc:1514] Processing DeleteTablet for tablet 591af678b6d74272a61f9db3d2ed9db3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:45:11 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:42248
I20260926 16:45:32.028548 5258 tablet_replica.cc:330] stopping tablet replica
I20260926 16:45:32.028833 5258 raft_consensus.cc:2229] T 591af678b6d74272a61f9db3d2ed9db3 P 9847782936414dd38e658a1faf3d2234 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:45:32.029075 5258 raft_consensus.cc:2258] T 591af678b6d74272a61f9db3d2ed9db3 P 9847782936414dd38e658a1faf3d2234 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:45:32.030049 5258 ts_tablet_manager.cc:1905] T 591af678b6d74272a61f9db3d2ed9db3 P 9847782936414dd38e658a1faf3d2234: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:45:32.033448 5258 ts_tablet_manager.cc:1918] T 591af678b6d74272a61f9db3d2ed9db3 P 9847782936414dd38e658a1faf3d2234: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:45:32.033525 5258 log.cc:1196] T 591af678b6d74272a61f9db3d2ed9db3 P 9847782936414dd38e658a1faf3d2234: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-0/wal/wals/591af678b6d74272a61f9db3d2ed9db3
I20260926 16:45:32.033833 5258 ts_tablet_manager.cc:1939] T 591af678b6d74272a61f9db3d2ed9db3 P 9847782936414dd38e658a1faf3d2234: Deleting consensus metadata
I20260926 16:45:32.035356 5043 catalog_manager.cc:4841] TS 9847782936414dd38e658a1faf3d2234 (127.21.12.65:34405): tablet 591af678b6d74272a61f9db3d2ed9db3 (table test-table [id=84d3f148ccaa4833952ef0c257e4948a]) successfully deleted
W20260926 16:45:32.112470 5254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:45:32.112696 5254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:45:32.112736 5254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:45:32.116721 5254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:45:32.116775 5254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:45:32.116874 5254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.66
I20260926 16:45:32.122303 5254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:45327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab
--principal=kudu/127.21.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.66:35041
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.66
--webserver_port=44039
--tserver_master_addrs=127.21.12.126:35725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.66
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:45:32.123291 5254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:45:32.124295 5254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:45:32.133617 5266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:45:32.133631 5264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:45:32.133631 5263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:45:32.133682 5254 server_base.cc:1034] running on GCE node
I20260926 16:45:32.134259 5254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:45:32.134856 5254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:45:32.136035 5254 hybrid_clock.cc:648] HybridClock initialized: now 1790441132136020 us; error 27 us; skew 500 ppm
Sep 26 16:45:32 dist-test-slave-n1hw krb5kdc[4473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441132, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:45:32.138801 5254 init.cc:377] Logged in from keytab as kudu/127.21.12.66@KRBTEST.COM (short username kudu)
I20260926 16:45:32.139571 5254 webserver.cc:460] Webserver started at http://127.21.12.66:44039/ using document root <none> and password file <none>
I20260926 16:45:32.140146 5254 fs_manager.cc:359] Metadata directory not provided
I20260926 16:45:32.140216 5254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:45:32.143914 5254 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:45:32.145874 5273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:45:32.147068 5254 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:45:32.147197 5254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-1/wal
uuid: "763e6b4243744af68be266b9d09c1aa7"
format_stamp: "Formatted at 2026-09-26 16:45:11 on dist-test-slave-n1hw"
server_key: "d9f8d6c1f50a439755e30ec80df2fe0d"
server_key_iv: "6b6111300153a74f160696fde1c3cc38"
server_key_version: "encryptionkey@0"
I20260926 16:45:32.147722 5254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:45:32.169675 5254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:45:32.172999 5254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:45:32.173197 5254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:45:32.173781 5254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:45:32.174911 5280 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260926 16:45:32.178561 5280 ts_tablet_manager.cc:1852] T 591af678b6d74272a61f9db3d2ed9db3 P 763e6b4243744af68be266b9d09c1aa7: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260926 16:45:32.178934 5280 ts_tablet_manager.cc:1905] T 591af678b6d74272a61f9db3d2ed9db3 P 763e6b4243744af68be266b9d09c1aa7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:45:32.182595 5280 ts_tablet_manager.cc:1918] T 591af678b6d74272a61f9db3d2ed9db3 P 763e6b4243744af68be266b9d09c1aa7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:45:32.182772 5280 ts_tablet_manager.cc:1939] T 591af678b6d74272a61f9db3d2ed9db3 P 763e6b4243744af68be266b9d09c1aa7: Deleting consensus metadata
I20260926 16:45:32.183550 5254 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260926 16:45:32.183728 5254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.000s
I20260926 16:45:32.183827 5254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:45:32.183878 5254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:45:32.192801 5254 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.66:35041
I20260926 16:45:32.192870 5388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.66:35041 every 8 connection(s)
I20260926 16:45:32.193673 5254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
Sep 26 16:45:32 dist-test-slave-n1hw krb5kdc[4473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441132, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.66@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:45:32.203033 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 5254
I20260926 16:45:32.203485 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.67:39145
--local_ip_for_outbound_sockets=127.21.12.67
--tserver_master_addrs=127.21.12.126:35725
--webserver_port=33883
--webserver_interface=127.21.12.67
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab
--principal=kudu/127.21.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:45327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:45:32.207197 5251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:32.195560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:60077 (local address 127.21.12.126:35725)
0926 16:45:32.195700 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 16:45:32.195703 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:45:32.196564 (+ 861us) server_negotiation.cc:407] Connection header received
0926 16:45:32.197484 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:45:32.197487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:45:32.197547 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:45:32.197648 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:45:32.198806 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:32.199338 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:32.200483 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:32.200721 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:32.200831 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:32.203359 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:45:32.203363 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:45:32.203365 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:45:32.203388 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:45:32.204922 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:45:32.205565 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:45:32.205575 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:45:32.205577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:45:32.205654 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:45:32.206052 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:45:32.206055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:45:32.206057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:45:32.206236 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:45:32.206354 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 16:45:32.206900 (+ 546us) server_negotiation.cc:299] Negotiation successful
0926 16:45:32.207030 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 16:45:32.207989 5391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:32.195976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.66:60077)
0926 16:45:32.196422 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:32.196463 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 16:45:32.197266 (+ 803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:45:32.197845 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:45:32.197852 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:45:32.198187 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:45:32.198651 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:32.198664 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:32.199466 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:32.199470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:32.200348 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:32.200357 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:32.200867 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:32.200871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:32.201040 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:32.201634 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:45:32.201651 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:45:32.203194 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:45:32.205090 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:45:32.205099 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:45:32.205116 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:45:32.205435 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:45:32.205787 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:45:32.205792 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:45:32.205794 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:45:32.205940 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:45:32.206353 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:45:32.206359 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:45:32.206616 (+ 257us) client_negotiation.cc:769] Sending connection context
0926 16:45:32.206846 (+ 230us) client_negotiation.cc:240] Negotiation successful
0926 16:45:32.207153 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":116,"threads_started":1}
I20260926 16:45:32.209271 5389 heartbeater.cc:347] Connected to a master server at 127.21.12.126:35725
I20260926 16:45:32.209499 5389 heartbeater.cc:464] Registering TS with master...
I20260926 16:45:32.210019 5389 heartbeater.cc:511] Master 127.21.12.126:35725 requested a full tablet report, sending...
I20260926 16:45:32.211091 5059 ts_manager.cc:194] Registered new tserver with Master: 763e6b4243744af68be266b9d09c1aa7 (127.21.12.66:35041)
I20260926 16:45:32.211961 5059 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.66@KRBTEST.COM'} at 127.21.12.66:60077
W20260926 16:45:32.315985 5392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:45:32.316246 5392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:45:32.316289 5392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:45:32.320367 5392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:45:32.320422 5392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:45:32.320482 5392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.67
I20260926 16:45:32.325568 5392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:45327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab
--principal=kudu/127.21.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.67:39145
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.67
--webserver_port=33883
--tserver_master_addrs=127.21.12.126:35725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.67
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:45:32.326578 5392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:45:32.327544 5392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:45:32.337050 5400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:45:32.337170 5392 server_base.cc:1034] running on GCE node
W20260926 16:45:32.337060 5397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:45:32.337354 5398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:45:32.337898 5392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:45:32.338516 5392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:45:32.339679 5392 hybrid_clock.cc:648] HybridClock initialized: now 1790441132339625 us; error 66 us; skew 500 ppm
Sep 26 16:45:32 dist-test-slave-n1hw krb5kdc[4473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441132, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:45:32.342478 5392 init.cc:377] Logged in from keytab as kudu/127.21.12.67@KRBTEST.COM (short username kudu)
I20260926 16:45:32.343340 5392 webserver.cc:460] Webserver started at http://127.21.12.67:33883/ using document root <none> and password file <none>
I20260926 16:45:32.343907 5392 fs_manager.cc:359] Metadata directory not provided
I20260926 16:45:32.344033 5392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:45:32.347769 5392 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260926 16:45:32.349963 5407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:45:32.351356 5392 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 16:45:32.351584 5392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-2/wal
uuid: "ed20749a2d644876849e5f81fe7bd137"
format_stamp: "Formatted at 2026-09-26 16:45:11 on dist-test-slave-n1hw"
server_key: "f690a34952386bf16d25b145cbf3cd10"
server_key_iv: "4d36fb9263f7c3e3d48a57ee33948462"
server_key_version: "encryptionkey@0"
I20260926 16:45:32.352173 5392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:45:32.367506 5392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:45:32.370373 5392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:45:32.370572 5392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:45:32.371127 5392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:45:32.371899 5392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:45:32.371969 5392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:45:32.372040 5392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:45:32.372090 5392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:45:32.380977 5392 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.67:39145
I20260926 16:45:32.380999 5521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.67:39145 every 8 connection(s)
I20260926 16:45:32.381948 5392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
I20260926 16:45:32.390070 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 5392
Sep 26 16:45:32 dist-test-slave-n1hw krb5kdc[4473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441132, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.67@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:45:32.396267 5251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:32.383733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:45573 (local address 127.21.12.126:35725)
0926 16:45:32.383889 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 16:45:32.383893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:45:32.384775 (+ 882us) server_negotiation.cc:407] Connection header received
0926 16:45:32.386231 (+ 1456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:45:32.386234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:45:32.386314 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:45:32.386395 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:45:32.387529 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:32.388063 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:32.389169 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:32.389415 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:32.389530 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:32.392415 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:45:32.392419 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:45:32.392421 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:45:32.392453 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:45:32.394012 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:45:32.394586 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:45:32.394590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:45:32.394594 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:45:32.394705 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:45:32.395082 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:45:32.395088 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:45:32.395091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:45:32.395321 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:45:32.395429 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 16:45:32.395932 (+ 503us) server_negotiation.cc:299] Negotiation successful
0926 16:45:32.396060 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 16:45:32.397138 5524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:32.384108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.67:45573)
0926 16:45:32.384567 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:32.384612 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 16:45:32.386027 (+ 1415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:45:32.386579 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:45:32.386586 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:45:32.386925 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:45:32.387381 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:32.387393 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:32.388185 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:32.388189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:32.389043 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:32.389052 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:32.389565 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:32.389568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:32.389738 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:32.390507 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:45:32.390532 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:45:32.392252 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:45:32.394148 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:45:32.394154 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:45:32.394167 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:45:32.394478 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:45:32.394807 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:45:32.394809 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:45:32.394811 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:45:32.394977 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:45:32.395426 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:45:32.395431 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:45:32.395632 (+ 201us) client_negotiation.cc:769] Sending connection context
0926 16:45:32.395864 (+ 232us) client_negotiation.cc:240] Negotiation successful
0926 16:45:32.396215 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":109,"threads_started":1}
I20260926 16:45:32.398428 5522 heartbeater.cc:347] Connected to a master server at 127.21.12.126:35725
I20260926 16:45:32.398659 5522 heartbeater.cc:464] Registering TS with master...
I20260926 16:45:32.398922 5525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:32.390728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57092 (local address 127.21.12.126:35725)
0926 16:45:32.390942 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 16:45:32.390945 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:45:32.391196 (+ 251us) server_negotiation.cc:407] Connection header received
0926 16:45:32.391336 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:45:32.391339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:45:32.391392 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:45:32.391492 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:45:32.392255 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:32.392883 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:32.393734 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:32.393994 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:32.394105 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:32.395590 (+ 1485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:45:32.395594 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:45:32.395597 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:45:32.395626 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:45:32.397162 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:45:32.397647 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:45:32.397650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:45:32.397651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:45:32.397738 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:45:32.398112 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:45:32.398115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:45:32.398116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:45:32.398269 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:45:32.398474 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 16:45:32.398650 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 16:45:32.398766 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":83,"threads_started":1}
I20260926 16:45:32.399199 5522 heartbeater.cc:511] Master 127.21.12.126:35725 requested a full tablet report, sending...
I20260926 16:45:32.400288 5058 ts_manager.cc:194] Registered new tserver with Master: ed20749a2d644876849e5f81fe7bd137 (127.21.12.67:39145)
I20260926 16:45:32.400892 5058 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.67@KRBTEST.COM'} at 127.21.12.67:45573
I20260926 16:45:32.412231 21553 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:45:32.415867 5525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:32.412806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57108 (local address 127.21.12.126:35725)
0926 16:45:32.412920 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 16:45:32.412923 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:45:32.413089 (+ 166us) server_negotiation.cc:407] Connection header received
0926 16:45:32.413210 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:45:32.413212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:45:32.413297 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:45:32.413379 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:45:32.413561 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:32.414123 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:32.414935 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:32.415152 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:32.415231 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:32.415477 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:45:32.415597 (+ 120us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 16:45:32.415784 (+ 187us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57108: token expired
Metrics: {"server-negotiator.queue_time_us":42}
W20260926 16:45:32.415969 5525 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57108: token expired
W20260926 16:45:32.416290 4988 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.21.12.126:35725)
I20260926 16:45:32.423519 5525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:32.416624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57124 (local address 127.21.12.126:35725)
0926 16:45:32.416736 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 16:45:32.416739 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:45:32.416750 (+ 11us) server_negotiation.cc:407] Connection header received
0926 16:45:32.416809 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:45:32.416811 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:45:32.416841 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:45:32.416907 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:45:32.417447 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:32.418009 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:32.418775 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:32.419018 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:32.419104 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:32.420391 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:45:32.420394 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:45:32.420396 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:45:32.420416 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:45:32.422012 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:45:32.422382 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:45:32.422385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:45:32.422386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:45:32.422458 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:45:32.422678 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:45:32.422680 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:45:32.422681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:45:32.422865 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:45:32.423012 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 16:45:32.423250 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 16:45:32.423363 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 16:45:32.426008 5058 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57124:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:45:32.429051 5058 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:45:32.436456 5176 tablet_service.cc:1467] Processing CreateTablet for tablet c9e026a50abe4c5f899132e9dbcff7cf (DEFAULT_TABLE table=test-table [id=7622c6c75cb547928ebf593621b1fbc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:45:32.437111 5176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9e026a50abe4c5f899132e9dbcff7cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:45:32.439771 5139 tablet_bootstrap.cc:492] T c9e026a50abe4c5f899132e9dbcff7cf P 9847782936414dd38e658a1faf3d2234: Bootstrap starting.
I20260926 16:45:32.440304 5256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:32.435388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.67:39145 (local address 127.0.0.1:45992)
0926 16:45:32.435509 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:32.435524 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:45:32.435601 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:45:32.436504 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:45:32.436507 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:45:32.436523 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:45:32.436647 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:32.436655 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:32.437890 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:32.437896 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:32.438718 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:32.438727 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:32.439779 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:32.439782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:32.439985 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:32.440036 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 16:45:32.440099 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 16:45:32.440182 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 16:45:32.440678 5139 tablet_bootstrap.cc:654] T c9e026a50abe4c5f899132e9dbcff7cf P 9847782936414dd38e658a1faf3d2234: Neither blocks nor log segments found. Creating new log.
I20260926 16:45:32.441054 5531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:32.435566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45992 (local address 127.21.12.67:39145)
0926 16:45:32.435975 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0926 16:45:32.435985 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:45:32.436024 (+ 39us) server_negotiation.cc:407] Connection header received
0926 16:45:32.436112 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:45:32.436118 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:45:32.436365 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:45:32.436531 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:45:32.436748 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:32.437718 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:32.438856 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:32.439649 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:32.440646 (+ 997us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:32.440815 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 16:45:32.440871 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 16:45:32.440947 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":182,"threads_started":1}
I20260926 16:45:32.442093 5139 tablet_bootstrap.cc:492] T c9e026a50abe4c5f899132e9dbcff7cf P 9847782936414dd38e658a1faf3d2234: No bootstrap required, opened a new log
I20260926 16:45:32.442202 5139 ts_tablet_manager.cc:1397] T c9e026a50abe4c5f899132e9dbcff7cf P 9847782936414dd38e658a1faf3d2234: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 16:45:32.442605 5139 raft_consensus.cc:348] T c9e026a50abe4c5f899132e9dbcff7cf P 9847782936414dd38e658a1faf3d2234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20749a2d644876849e5f81fe7bd137" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 39145 } } peers { permanent_uuid: "763e6b4243744af68be266b9d09c1aa7" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 35041 } } peers { permanent_uuid: "9847782936414dd38e658a1faf3d2234" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 34405 } }
I20260926 16:45:32.442721 5139 raft_consensus.cc:374] T c9e026a50abe4c5f899132e9dbcff7cf P 9847782936414dd38e658a1faf3d2234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:45:32.442772 5139 raft_consensus.cc:729] T c9e026a50abe4c5f899132e9dbcff7cf P 9847782936414dd38e658a1faf3d2234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9847782936414dd38e658a1faf3d2234, State: Initialized, Role: FOLLOWER
I20260926 16:45:32.442924 5139 consensus_queue.cc:260] T c9e026a50abe4c5f899132e9dbcff7cf P 9847782936414dd38e658a1faf3d2234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20749a2d644876849e5f81fe7bd137" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 39145 } } peers { permanent_uuid: "763e6b4243744af68be266b9d09c1aa7" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 35041 } } peers { permanent_uuid: "9847782936414dd38e658a1faf3d2234" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 34405 } }
I20260926 16:45:32.443197 5139 ts_tablet_manager.cc:1428] T c9e026a50abe4c5f899132e9dbcff7cf P 9847782936414dd38e658a1faf3d2234: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 16:45:32.443799 5455 tablet_service.cc:1467] Processing CreateTablet for tablet c9e026a50abe4c5f899132e9dbcff7cf (DEFAULT_TABLE table=test-table [id=7622c6c75cb547928ebf593621b1fbc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:45:32.444929 5455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9e026a50abe4c5f899132e9dbcff7cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:45:32.445207 5532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:32.435877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:35041 (local address 127.0.0.1:35892)
0926 16:45:32.436120 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:32.436133 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 16:45:32.436214 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:45:32.437603 (+ 1389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:45:32.437606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:45:32.437628 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:45:32.437724 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:32.437734 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:32.439059 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:32.439062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:32.440668 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:32.440678 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:32.444794 (+ 4116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:32.444798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:32.444885 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:32.444914 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 16:45:32.445000 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 16:45:32.445066 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":102,"threads_started":1}
I20260926 16:45:32.445297 5533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:32.436300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35892 (local address 127.21.12.66:35041)
0926 16:45:32.436949 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0926 16:45:32.436960 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:45:32.436982 (+ 22us) server_negotiation.cc:407] Connection header received
0926 16:45:32.437057 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:45:32.437061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:45:32.437267 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:45:32.437431 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:45:32.437886 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:32.438935 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:32.441521 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:32.444664 (+ 3143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:32.444791 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:32.444925 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 16:45:32.445038 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 16:45:32.445141 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":172,"threads_started":1}
I20260926 16:45:32.447091 5323 tablet_service.cc:1467] Processing CreateTablet for tablet c9e026a50abe4c5f899132e9dbcff7cf (DEFAULT_TABLE table=test-table [id=7622c6c75cb547928ebf593621b1fbc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:45:32.448006 5323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9e026a50abe4c5f899132e9dbcff7cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:45:32.451478 5280 tablet_bootstrap.cc:492] T c9e026a50abe4c5f899132e9dbcff7cf P 763e6b4243744af68be266b9d09c1aa7: Bootstrap starting.
I20260926 16:45:32.451982 5535 tablet_bootstrap.cc:492] T c9e026a50abe4c5f899132e9dbcff7cf P ed20749a2d644876849e5f81fe7bd137: Bootstrap starting.
I20260926 16:45:32.453679 5280 tablet_bootstrap.cc:654] T c9e026a50abe4c5f899132e9dbcff7cf P 763e6b4243744af68be266b9d09c1aa7: Neither blocks nor log segments found. Creating new log.
I20260926 16:45:32.453696 5535 tablet_bootstrap.cc:654] T c9e026a50abe4c5f899132e9dbcff7cf P ed20749a2d644876849e5f81fe7bd137: Neither blocks nor log segments found. Creating new log.
I20260926 16:45:32.454558 5280 log.cc:824] T c9e026a50abe4c5f899132e9dbcff7cf P 763e6b4243744af68be266b9d09c1aa7: Log is configured to *not* fsync() on all Append() calls
I20260926 16:45:32.454959 5535 log.cc:824] T c9e026a50abe4c5f899132e9dbcff7cf P ed20749a2d644876849e5f81fe7bd137: Log is configured to *not* fsync() on all Append() calls
I20260926 16:45:32.456398 5280 tablet_bootstrap.cc:492] T c9e026a50abe4c5f899132e9dbcff7cf P 763e6b4243744af68be266b9d09c1aa7: No bootstrap required, opened a new log
I20260926 16:45:32.456576 5280 ts_tablet_manager.cc:1397] T c9e026a50abe4c5f899132e9dbcff7cf P 763e6b4243744af68be266b9d09c1aa7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 16:45:32.456749 5535 tablet_bootstrap.cc:492] T c9e026a50abe4c5f899132e9dbcff7cf P ed20749a2d644876849e5f81fe7bd137: No bootstrap required, opened a new log
I20260926 16:45:32.456954 5535 ts_tablet_manager.cc:1397] T c9e026a50abe4c5f899132e9dbcff7cf P ed20749a2d644876849e5f81fe7bd137: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260926 16:45:32.459642 5280 raft_consensus.cc:348] T c9e026a50abe4c5f899132e9dbcff7cf P 763e6b4243744af68be266b9d09c1aa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20749a2d644876849e5f81fe7bd137" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 39145 } } peers { permanent_uuid: "763e6b4243744af68be266b9d09c1aa7" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 35041 } } peers { permanent_uuid: "9847782936414dd38e658a1faf3d2234" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 34405 } }
I20260926 16:45:32.459975 5280 raft_consensus.cc:374] T c9e026a50abe4c5f899132e9dbcff7cf P 763e6b4243744af68be266b9d09c1aa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:45:32.459789 5535 raft_consensus.cc:348] T c9e026a50abe4c5f899132e9dbcff7cf P ed20749a2d644876849e5f81fe7bd137 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20749a2d644876849e5f81fe7bd137" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 39145 } } peers { permanent_uuid: "763e6b4243744af68be266b9d09c1aa7" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 35041 } } peers { permanent_uuid: "9847782936414dd38e658a1faf3d2234" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 34405 } }
I20260926 16:45:32.460039 5280 raft_consensus.cc:729] T c9e026a50abe4c5f899132e9dbcff7cf P 763e6b4243744af68be266b9d09c1aa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 763e6b4243744af68be266b9d09c1aa7, State: Initialized, Role: FOLLOWER
I20260926 16:45:32.460068 5535 raft_consensus.cc:374] T c9e026a50abe4c5f899132e9dbcff7cf P ed20749a2d644876849e5f81fe7bd137 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:45:32.460135 5535 raft_consensus.cc:729] T c9e026a50abe4c5f899132e9dbcff7cf P ed20749a2d644876849e5f81fe7bd137 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed20749a2d644876849e5f81fe7bd137, State: Initialized, Role: FOLLOWER
I20260926 16:45:32.460498 5280 consensus_queue.cc:260] T c9e026a50abe4c5f899132e9dbcff7cf P 763e6b4243744af68be266b9d09c1aa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20749a2d644876849e5f81fe7bd137" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 39145 } } peers { permanent_uuid: "763e6b4243744af68be266b9d09c1aa7" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 35041 } } peers { permanent_uuid: "9847782936414dd38e658a1faf3d2234" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 34405 } }
I20260926 16:45:32.460530 5535 consensus_queue.cc:260] T c9e026a50abe4c5f899132e9dbcff7cf P ed20749a2d644876849e5f81fe7bd137 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20749a2d644876849e5f81fe7bd137" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 39145 } } peers { permanent_uuid: "763e6b4243744af68be266b9d09c1aa7" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 35041 } } peers { permanent_uuid: "9847782936414dd38e658a1faf3d2234" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 34405 } }
I20260926 16:45:32.461107 5522 heartbeater.cc:503] Master 127.21.12.126:35725 was elected leader, sending a full tablet report...
I20260926 16:45:32.461377 5535 ts_tablet_manager.cc:1428] T c9e026a50abe4c5f899132e9dbcff7cf P ed20749a2d644876849e5f81fe7bd137: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260926 16:45:32.461436 5280 ts_tablet_manager.cc:1428] T c9e026a50abe4c5f899132e9dbcff7cf P 763e6b4243744af68be266b9d09c1aa7: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 16:45:32.462020 5389 heartbeater.cc:503] Master 127.21.12.126:35725 was elected leader, sending a full tablet report...
W20260926 16:45:32.488173 5250 tablet.cc:2369] T c9e026a50abe4c5f899132e9dbcff7cf P 9847782936414dd38e658a1faf3d2234: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:45:32.632890 5523 tablet.cc:2369] T c9e026a50abe4c5f899132e9dbcff7cf P ed20749a2d644876849e5f81fe7bd137: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:45:32.694772 5390 tablet.cc:2369] T c9e026a50abe4c5f899132e9dbcff7cf P 763e6b4243744af68be266b9d09c1aa7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:45:32.772688 5538 raft_consensus.cc:482] T c9e026a50abe4c5f899132e9dbcff7cf P ed20749a2d644876849e5f81fe7bd137 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:45:32.772917 5538 raft_consensus.cc:504] T c9e026a50abe4c5f899132e9dbcff7cf P ed20749a2d644876849e5f81fe7bd137 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20749a2d644876849e5f81fe7bd137" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 39145 } } peers { permanent_uuid: "763e6b4243744af68be266b9d09c1aa7" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 35041 } } peers { permanent_uuid: "9847782936414dd38e658a1faf3d2234" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 34405 } }
I20260926 16:45:32.774117 5538 leader_election.cc:288] T c9e026a50abe4c5f899132e9dbcff7cf P ed20749a2d644876849e5f81fe7bd137 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 763e6b4243744af68be266b9d09c1aa7 (127.21.12.66:35041), 9847782936414dd38e658a1faf3d2234 (127.21.12.65:34405)
I20260926 16:45:32.778553 5524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:32.774334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:35041 (local address 127.21.12.67:59325)
0926 16:45:32.774436 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:32.774452 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:45:32.774549 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:45:32.774821 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:45:32.774827 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:45:32.774851 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:45:32.774984 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:32.774991 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:32.775953 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:32.775956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:32.777203 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:32.777214 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:32.778107 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:32.778110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:32.778225 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:32.778246 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 16:45:32.778310 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 16:45:32.778411 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260926 16:45:32.778591 5533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:32.774448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:59325 (local address 127.21.12.66:35041)
0926 16:45:32.774565 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 16:45:32.774570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:45:32.774590 (+ 20us) server_negotiation.cc:407] Connection header received
0926 16:45:32.774650 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:45:32.774652 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:45:32.774701 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:45:32.774775 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:45:32.775105 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:32.775834 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:32.777347 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:32.777925 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:32.778048 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:32.778098 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 16:45:32.778362 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 16:45:32.778476 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 16:45:32.778760 5540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:32.774442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:34405 (local address 127.21.12.67:54517)
0926 16:45:32.774739 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:32.774775 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 16:45:32.774901 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:45:32.775373 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:45:32.775376 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:45:32.775391 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:45:32.775474 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:32.775478 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:32.776380 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:32.776383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:32.777472 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:32.777479 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:32.778387 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:32.778390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:32.778476 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:32.778498 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 16:45:32.778556 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 16:45:32.778622 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":102,"threads_started":1}
I20260926 16:45:32.778864 5541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:32.774743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:54517 (local address 127.21.12.65:34405)
0926 16:45:32.775066 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0926 16:45:32.775071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:45:32.775085 (+ 14us) server_negotiation.cc:407] Connection header received
0926 16:45:32.775131 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:45:32.775134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:45:32.775194 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:45:32.775328 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:45:32.775588 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:32.776268 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:32.777597 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:32.778251 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:32.778417 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:32.778611 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 16:45:32.778656 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 16:45:32.778739 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":69,"threads_started":1}
I20260926 16:45:32.779402 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e026a50abe4c5f899132e9dbcff7cf" candidate_uuid: "ed20749a2d644876849e5f81fe7bd137" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "763e6b4243744af68be266b9d09c1aa7" is_pre_election: true
I20260926 16:45:32.779546 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e026a50abe4c5f899132e9dbcff7cf" candidate_uuid: "ed20749a2d644876849e5f81fe7bd137" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9847782936414dd38e658a1faf3d2234" is_pre_election: true
I20260926 16:45:32.779768 5343 raft_consensus.cc:2454] T c9e026a50abe4c5f899132e9dbcff7cf P 763e6b4243744af68be266b9d09c1aa7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed20749a2d644876849e5f81fe7bd137 in term 0.
I20260926 16:45:32.779893 5201 raft_consensus.cc:2454] T c9e026a50abe4c5f899132e9dbcff7cf P 9847782936414dd38e658a1faf3d2234 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed20749a2d644876849e5f81fe7bd137 in term 0.
I20260926 16:45:32.780320 5408 leader_election.cc:302] T c9e026a50abe4c5f899132e9dbcff7cf P ed20749a2d644876849e5f81fe7bd137 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9847782936414dd38e658a1faf3d2234, ed20749a2d644876849e5f81fe7bd137; no voters:
I20260926 16:45:32.780614 5538 raft_consensus.cc:2781] T c9e026a50abe4c5f899132e9dbcff7cf P ed20749a2d644876849e5f81fe7bd137 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:45:32.780697 5538 raft_consensus.cc:482] T c9e026a50abe4c5f899132e9dbcff7cf P ed20749a2d644876849e5f81fe7bd137 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:45:32.780740 5538 raft_consensus.cc:3037] T c9e026a50abe4c5f899132e9dbcff7cf P ed20749a2d644876849e5f81fe7bd137 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:45:32.781728 5538 raft_consensus.cc:504] T c9e026a50abe4c5f899132e9dbcff7cf P ed20749a2d644876849e5f81fe7bd137 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20749a2d644876849e5f81fe7bd137" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 39145 } } peers { permanent_uuid: "763e6b4243744af68be266b9d09c1aa7" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 35041 } } peers { permanent_uuid: "9847782936414dd38e658a1faf3d2234" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 34405 } }
I20260926 16:45:32.782069 5538 leader_election.cc:288] T c9e026a50abe4c5f899132e9dbcff7cf P ed20749a2d644876849e5f81fe7bd137 [CANDIDATE]: Term 1 election: Requested vote from peers 763e6b4243744af68be266b9d09c1aa7 (127.21.12.66:35041), 9847782936414dd38e658a1faf3d2234 (127.21.12.65:34405)
I20260926 16:45:32.782370 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e026a50abe4c5f899132e9dbcff7cf" candidate_uuid: "ed20749a2d644876849e5f81fe7bd137" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "763e6b4243744af68be266b9d09c1aa7"
I20260926 16:45:32.782418 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e026a50abe4c5f899132e9dbcff7cf" candidate_uuid: "ed20749a2d644876849e5f81fe7bd137" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9847782936414dd38e658a1faf3d2234"
I20260926 16:45:32.782505 5343 raft_consensus.cc:3037] T c9e026a50abe4c5f899132e9dbcff7cf P 763e6b4243744af68be266b9d09c1aa7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:45:32.782545 5201 raft_consensus.cc:3037] T c9e026a50abe4c5f899132e9dbcff7cf P 9847782936414dd38e658a1faf3d2234 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:45:32.783502 5201 raft_consensus.cc:2454] T c9e026a50abe4c5f899132e9dbcff7cf P 9847782936414dd38e658a1faf3d2234 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed20749a2d644876849e5f81fe7bd137 in term 1.
I20260926 16:45:32.783519 5343 raft_consensus.cc:2454] T c9e026a50abe4c5f899132e9dbcff7cf P 763e6b4243744af68be266b9d09c1aa7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed20749a2d644876849e5f81fe7bd137 in term 1.
I20260926 16:45:32.783767 5408 leader_election.cc:302] T c9e026a50abe4c5f899132e9dbcff7cf P ed20749a2d644876849e5f81fe7bd137 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9847782936414dd38e658a1faf3d2234, ed20749a2d644876849e5f81fe7bd137; no voters:
I20260926 16:45:32.783957 5538 raft_consensus.cc:2781] T c9e026a50abe4c5f899132e9dbcff7cf P ed20749a2d644876849e5f81fe7bd137 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:45:32.784137 5538 raft_consensus.cc:686] T c9e026a50abe4c5f899132e9dbcff7cf P ed20749a2d644876849e5f81fe7bd137 [term 1 LEADER]: Becoming Leader. State: Replica: ed20749a2d644876849e5f81fe7bd137, State: Running, Role: LEADER
I20260926 16:45:32.784497 5538 consensus_queue.cc:237] T c9e026a50abe4c5f899132e9dbcff7cf P ed20749a2d644876849e5f81fe7bd137 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20749a2d644876849e5f81fe7bd137" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 39145 } } peers { permanent_uuid: "763e6b4243744af68be266b9d09c1aa7" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 35041 } } peers { permanent_uuid: "9847782936414dd38e658a1faf3d2234" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 34405 } }
I20260926 16:45:32.787897 5059 catalog_manager.cc:5495] T c9e026a50abe4c5f899132e9dbcff7cf P ed20749a2d644876849e5f81fe7bd137 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed20749a2d644876849e5f81fe7bd137 (127.21.12.67). New cstate: current_term: 1 leader_uuid: "ed20749a2d644876849e5f81fe7bd137" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20749a2d644876849e5f81fe7bd137" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 39145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "763e6b4243744af68be266b9d09c1aa7" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 35041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9847782936414dd38e658a1faf3d2234" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 34405 } health_report { overall_health: UNKNOWN } } }
I20260926 16:45:32.806998 5531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:32.803234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46008 (local address 127.21.12.67:39145)
0926 16:45:32.803371 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 16:45:32.803375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:45:32.803388 (+ 13us) server_negotiation.cc:407] Connection header received
0926 16:45:32.803487 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:45:32.803489 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:45:32.803537 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:45:32.803619 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:45:32.803871 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:32.804381 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:32.805527 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:32.805738 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:32.805873 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:32.806159 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:45:32.806534 (+ 375us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:45:32.806622 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 16:45:32.806717 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 16:45:32.806765 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 16:45:32.813158 5343 raft_consensus.cc:1261] T c9e026a50abe4c5f899132e9dbcff7cf P 763e6b4243744af68be266b9d09c1aa7 [term 1 FOLLOWER]: Refusing update from remote peer ed20749a2d644876849e5f81fe7bd137: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:45:32.813158 5201 raft_consensus.cc:1261] T c9e026a50abe4c5f899132e9dbcff7cf P 9847782936414dd38e658a1faf3d2234 [term 1 FOLLOWER]: Refusing update from remote peer ed20749a2d644876849e5f81fe7bd137: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:45:32.813889 5542 consensus_queue.cc:1035] T c9e026a50abe4c5f899132e9dbcff7cf P ed20749a2d644876849e5f81fe7bd137 [LEADER]: Connected to new peer: Peer: permanent_uuid: "763e6b4243744af68be266b9d09c1aa7" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 35041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:45:32.814085 5538 consensus_queue.cc:1035] T c9e026a50abe4c5f899132e9dbcff7cf P ed20749a2d644876849e5f81fe7bd137 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9847782936414dd38e658a1faf3d2234" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 34405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:45:32.821306 5546 mvcc.cc:205] Tried to move back new op lower bound from 7333646879994912768 to 7333646879887925248. Current Snapshot: MvccSnapshot[applied={T|T < 7333646879994912768}]
I20260926 16:45:32.828941 5059 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:45:32.829068 5059 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:45:32.831404 5059 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4: Sending DeleteTablet for 3 replicas of tablet c9e026a50abe4c5f899132e9dbcff7cf
I20260926 16:45:32.832122 5176 tablet_service.cc:1514] Processing DeleteTablet for tablet c9e026a50abe4c5f899132e9dbcff7cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:45:32 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:42248
I20260926 16:45:32.832090 5323 tablet_service.cc:1514] Processing DeleteTablet for tablet c9e026a50abe4c5f899132e9dbcff7cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:45:32 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:35892
I20260926 16:45:32.832485 5555 tablet_replica.cc:330] T c9e026a50abe4c5f899132e9dbcff7cf P 9847782936414dd38e658a1faf3d2234: stopping tablet replica
I20260926 16:45:32.832625 5555 raft_consensus.cc:2229] T c9e026a50abe4c5f899132e9dbcff7cf P 9847782936414dd38e658a1faf3d2234 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:45:32.832612 5455 tablet_service.cc:1514] Processing DeleteTablet for tablet c9e026a50abe4c5f899132e9dbcff7cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:45:32 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:45992
I20260926 16:45:32.832757 5555 raft_consensus.cc:2258] T c9e026a50abe4c5f899132e9dbcff7cf P 9847782936414dd38e658a1faf3d2234 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:45:32.833056 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 5097
I20260926 16:45:32.833113 5557 tablet_replica.cc:330] T c9e026a50abe4c5f899132e9dbcff7cf P ed20749a2d644876849e5f81fe7bd137: stopping tablet replica
I20260926 16:45:32.833384 5557 raft_consensus.cc:2229] T c9e026a50abe4c5f899132e9dbcff7cf P ed20749a2d644876849e5f81fe7bd137 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:45:32.833905 5557 raft_consensus.cc:2258] T c9e026a50abe4c5f899132e9dbcff7cf P ed20749a2d644876849e5f81fe7bd137 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:45:32.834913 5556 tablet_replica.cc:330] T c9e026a50abe4c5f899132e9dbcff7cf P 763e6b4243744af68be266b9d09c1aa7: stopping tablet replica
I20260926 16:45:32.835279 5556 raft_consensus.cc:2229] T c9e026a50abe4c5f899132e9dbcff7cf P 763e6b4243744af68be266b9d09c1aa7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:45:32.835433 5557 ts_tablet_manager.cc:1905] T c9e026a50abe4c5f899132e9dbcff7cf P ed20749a2d644876849e5f81fe7bd137: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:45:32.835680 5556 raft_consensus.cc:2258] T c9e026a50abe4c5f899132e9dbcff7cf P 763e6b4243744af68be266b9d09c1aa7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:45:32.838352 5557 ts_tablet_manager.cc:1918] T c9e026a50abe4c5f899132e9dbcff7cf P ed20749a2d644876849e5f81fe7bd137: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:45:32.838449 5557 log.cc:1196] T c9e026a50abe4c5f899132e9dbcff7cf P ed20749a2d644876849e5f81fe7bd137: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-2/wal/wals/c9e026a50abe4c5f899132e9dbcff7cf
I20260926 16:45:32.838815 5557 ts_tablet_manager.cc:1939] T c9e026a50abe4c5f899132e9dbcff7cf P ed20749a2d644876849e5f81fe7bd137: Deleting consensus metadata
I20260926 16:45:32.839975 5046 catalog_manager.cc:4841] TS ed20749a2d644876849e5f81fe7bd137 (127.21.12.67:39145): tablet c9e026a50abe4c5f899132e9dbcff7cf (table test-table [id=7622c6c75cb547928ebf593621b1fbc3]) successfully deleted
W20260926 16:45:32.840864 5043 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.12.65:34405 (error 108)
I20260926 16:45:32.841079 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 5254
I20260926 16:45:32.843422 5532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:32.843122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:34405 (local address 127.0.0.1:42254)
0926 16:45:32.843266 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:32.843338 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.65:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 16:45:32.843649 5043 catalog_manager.cc:4584] TS 9847782936414dd38e658a1faf3d2234 (127.21.12.65:34405): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c9e026a50abe4c5f899132e9dbcff7cf: Network error: Client connection negotiation failed: client connection to 127.21.12.65:34405: connect: Connection refused (error 111)
I20260926 16:45:32.847777 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 5392
I20260926 16:45:32.848768 5532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:32.847962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:35041 (local address 127.0.0.1:35902)
0926 16:45:32.848611 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:32.848677 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.66:35041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":552}
I20260926 16:45:32.853399 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 5026
I20260926 16:45:52.860870 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:35725
--webserver_interface=127.21.12.126
--webserver_port=41287
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:45327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:35725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:45:52.970600 5559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:45:52.970902 5559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:45:52.970945 5559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:45:52.974995 5559 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 16:45:52.975049 5559 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 16:45:52.975065 5559 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 16:45:52.975080 5559 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:45:52.975097 5559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:45:52.975150 5559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:45:52.975184 5559 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:45:52.975205 5559 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:45:52.980599 5559 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:45327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:35725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:35725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.126
--webserver_port=41287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:45:52.981685 5559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:45:52.982784 5559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:45:52.991816 5564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:45:52.991885 5567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:45:52.991839 5559 server_base.cc:1034] running on GCE node
W20260926 16:45:52.992041 5565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:45:52.992591 5559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:45:52.993676 5559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:45:52.994865 5559 hybrid_clock.cc:648] HybridClock initialized: now 1790441152994848 us; error 27 us; skew 500 ppm
Sep 26 16:45:52 dist-test-slave-n1hw krb5kdc[4473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441152, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:45:53.002444 5559 init.cc:377] Logged in from keytab as kudu/127.21.12.126@KRBTEST.COM (short username kudu)
I20260926 16:45:53.003379 5559 webserver.cc:460] Webserver started at http://127.21.12.126:41287/ using document root <none> and password file <none>
I20260926 16:45:53.003961 5559 fs_manager.cc:359] Metadata directory not provided
I20260926 16:45:53.004056 5559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:45:53.008088 5559 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:45:53.010293 5574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:45:53.011610 5559 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:45:53.011755 5559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "25a40b6cfa6544cdb74879e3751924c4"
format_stamp: "Formatted at 2026-09-26 16:45:10 on dist-test-slave-n1hw"
server_key: "7a3a695de4ee7cd939009ecc2d9e3f4e"
server_key_iv: "c6ba6db48d53dbb31bc22985e3583c26"
server_key_version: "encryptionkey@0"
I20260926 16:45:53.012297 5559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:45:53.028645 5559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:45:53.031550 5559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:45:53.031741 5559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:45:53.038834 5559 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.126:35725
I20260926 16:45:53.039116 5627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.126:35725 every 8 connection(s)
I20260926 16:45:53.039899 5559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:45:53.044066 5628 sys_catalog.cc:263] Verifying existing consensus state
I20260926 16:45:53.045661 5628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4: Bootstrap starting.
I20260926 16:45:53.047446 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 5559
I20260926 16:45:53.047915 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.65:34405
--local_ip_for_outbound_sockets=127.21.12.65
--tserver_master_addrs=127.21.12.126:35725
--webserver_port=32957
--webserver_interface=127.21.12.65
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab
--principal=kudu/127.21.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:45327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:45:53.063069 5628 log.cc:824] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4: Log is configured to *not* fsync() on all Append() calls
I20260926 16:45:53.074466 5628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 16:45:53.074854 5628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4: Bootstrap complete.
I20260926 16:45:53.078893 5628 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25a40b6cfa6544cdb74879e3751924c4" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 35725 } }
I20260926 16:45:53.079301 5628 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25a40b6cfa6544cdb74879e3751924c4, State: Initialized, Role: FOLLOWER
I20260926 16:45:53.079784 5628 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25a40b6cfa6544cdb74879e3751924c4" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 35725 } }
I20260926 16:45:53.079931 5628 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:45:53.080017 5628 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:45:53.080145 5628 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [term 2 FOLLOWER]: Advancing to term 3
I20260926 16:45:53.082046 5628 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25a40b6cfa6544cdb74879e3751924c4" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 35725 } }
I20260926 16:45:53.082392 5628 leader_election.cc:302] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25a40b6cfa6544cdb74879e3751924c4; no voters:
I20260926 16:45:53.082738 5628 leader_election.cc:288] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 16:45:53.082827 5632 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 16:45:53.083053 5632 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [term 3 LEADER]: Becoming Leader. State: Replica: 25a40b6cfa6544cdb74879e3751924c4, State: Running, Role: LEADER
I20260926 16:45:53.083416 5632 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25a40b6cfa6544cdb74879e3751924c4" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 35725 } }
I20260926 16:45:53.083997 5628 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:45:53.084810 5633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25a40b6cfa6544cdb74879e3751924c4. Latest consensus state: current_term: 3 leader_uuid: "25a40b6cfa6544cdb74879e3751924c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25a40b6cfa6544cdb74879e3751924c4" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 35725 } } }
I20260926 16:45:53.084957 5633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [sys.catalog]: This master's current role is: LEADER
I20260926 16:45:53.085383 5634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "25a40b6cfa6544cdb74879e3751924c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25a40b6cfa6544cdb74879e3751924c4" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 35725 } } }
I20260926 16:45:53.085559 5634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4 [sys.catalog]: This master's current role is: LEADER
I20260926 16:45:53.085875 5641 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:45:53.090916 5641 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:45:53.091634 5641 catalog_manager.cc:1259] Loaded cluster ID: 8099a53dc86f448086e3dea5e9ed0398
I20260926 16:45:53.091730 5641 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:45:53.094369 5641 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:45:53.095103 5641 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4: Loaded TSK: 0
I20260926 16:45:53.095186 5641 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 25a40b6cfa6544cdb74879e3751924c4: Loaded TSK: 1
F20260926 16:45:53.096376 5641 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd2e264d9ad google::LogMessage::Fail() at ??:0
@ 0x7fd2e265088a google::LogMessage::SendToLog() at ??:0
@ 0x7fd2e264d4de google::LogMessage::Flush() at ??:0
@ 0x7fd2e264ea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd2e531118d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fd2e5309f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fd2ecfb27bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fd2ecf8c99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fd2ecf8d178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fd2ecfcedf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fd2ecfed82b std::function<>::operator()() at ??:0
@ 0x7fd2ecf8cb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fd2ecf8df3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd2ecf8a579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fd2ecfce640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fd2e489f3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd2e489fc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fd2e48a13f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fd2e4890512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd2e5ed46db start_thread at ??:0
@ 0x7fd2e114b61f clone at ??:0
W20260926 16:45:53.206133 5630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:45:53.206396 5630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:45:53.206471 5630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:45:53.210587 5630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:45:53.210635 5630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:45:53.210722 5630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.65
I20260926 16:45:53.215874 5630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:45327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab
--principal=kudu/127.21.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.65:34405
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.65
--webserver_port=32957
--tserver_master_addrs=127.21.12.126:35725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.65
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:45:53.216964 5630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:45:53.218048 5630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:45:53.227596 5678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:45:53.227587 5675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:45:53.227880 5630 server_base.cc:1034] running on GCE node
W20260926 16:45:53.227632 5676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:45:53.228521 5630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:45:53.229171 5630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:45:53.230389 5630 hybrid_clock.cc:648] HybridClock initialized: now 1790441153230364 us; error 43 us; skew 500 ppm
Sep 26 16:45:53 dist-test-slave-n1hw krb5kdc[4473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441153, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:45:53.233242 5630 init.cc:377] Logged in from keytab as kudu/127.21.12.65@KRBTEST.COM (short username kudu)
I20260926 16:45:53.234205 5630 webserver.cc:460] Webserver started at http://127.21.12.65:32957/ using document root <none> and password file <none>
I20260926 16:45:53.234769 5630 fs_manager.cc:359] Metadata directory not provided
I20260926 16:45:53.234830 5630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:45:53.238737 5630 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:45:53.241168 5685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:45:53.242501 5630 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 16:45:53.242637 5630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-0/wal
uuid: "9847782936414dd38e658a1faf3d2234"
format_stamp: "Formatted at 2026-09-26 16:45:10 on dist-test-slave-n1hw"
server_key: "f95bbc156153ccc503343b53f20644e4"
server_key_iv: "2ed0b0449f353e1bf084a77ddb634014"
server_key_version: "encryptionkey@0"
I20260926 16:45:53.243175 5630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:45:53.269410 5630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:45:53.272811 5630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:45:53.273015 5630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:45:53.273604 5630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:45:53.274823 5692 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 16:45:53.278849 5630 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 16:45:53.278909 5630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260926 16:45:53.278956 5630 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 16:45:53.281190 5630 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 16:45:53.281256 5630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260926 16:45:53.281522 5692 tablet_bootstrap.cc:492] T c9e026a50abe4c5f899132e9dbcff7cf P 9847782936414dd38e658a1faf3d2234: Bootstrap starting.
I20260926 16:45:53.291857 5630 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.65:34405
I20260926 16:45:53.292194 5799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.65:34405 every 8 connection(s)
I20260926 16:45:53.293124 5630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
I20260926 16:45:53.295481 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 5630
I20260926 16:45:53.295974 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.66:35041
--local_ip_for_outbound_sockets=127.21.12.66
--tserver_master_addrs=127.21.12.126:35725
--webserver_port=44039
--webserver_interface=127.21.12.66
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab
--principal=kudu/127.21.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:45327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:45:53.302356 5692 log.cc:824] T c9e026a50abe4c5f899132e9dbcff7cf P 9847782936414dd38e658a1faf3d2234: Log is configured to *not* fsync() on all Append() calls
W20260926 16:45:53.308964 5802 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:45:53.295560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.65:33143)
0926 16:45:53.296898 (+ 1338us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:53.296955 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 16:45:53.298238 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:45:53.308003 (+ 9765us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1102,"thread_start_us":136,"threads_started":1}
W20260926 16:45:53.309535 5800 heartbeater.cc:644] Failed to heartbeat to 127.21.12.126:35725 (0 consecutive failures): Network error: Failed to ping master at 127.21.12.126:35725: Client connection negotiation failed: client connection to 127.21.12.126:35725: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 16:45:53.310262 5802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:53.310012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.65:50663)
0926 16:45:53.310134 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:53.310215 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 16:45:53.310998 5802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:53.310779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.65:49761)
0926 16:45:53.310897 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:53.310942 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 16:45:53.311208 5800 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 16:45:53.313079 5692 tablet_bootstrap.cc:492] T c9e026a50abe4c5f899132e9dbcff7cf P 9847782936414dd38e658a1faf3d2234: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 16:45:53.313503 5692 tablet_bootstrap.cc:492] T c9e026a50abe4c5f899132e9dbcff7cf P 9847782936414dd38e658a1faf3d2234: Bootstrap complete.
I20260926 16:45:53.314991 5692 ts_tablet_manager.cc:1397] T c9e026a50abe4c5f899132e9dbcff7cf P 9847782936414dd38e658a1faf3d2234: Time spent bootstrapping tablet: real 0.034s user 0.020s sys 0.011s
I20260926 16:45:53.318297 5692 raft_consensus.cc:348] T c9e026a50abe4c5f899132e9dbcff7cf P 9847782936414dd38e658a1faf3d2234 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20749a2d644876849e5f81fe7bd137" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 39145 } } peers { permanent_uuid: "763e6b4243744af68be266b9d09c1aa7" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 35041 } } peers { permanent_uuid: "9847782936414dd38e658a1faf3d2234" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 34405 } }
I20260926 16:45:53.318853 5692 raft_consensus.cc:729] T c9e026a50abe4c5f899132e9dbcff7cf P 9847782936414dd38e658a1faf3d2234 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9847782936414dd38e658a1faf3d2234, State: Initialized, Role: FOLLOWER
I20260926 16:45:53.319367 5692 consensus_queue.cc:260] T c9e026a50abe4c5f899132e9dbcff7cf P 9847782936414dd38e658a1faf3d2234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20749a2d644876849e5f81fe7bd137" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 39145 } } peers { permanent_uuid: "763e6b4243744af68be266b9d09c1aa7" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 35041 } } peers { permanent_uuid: "9847782936414dd38e658a1faf3d2234" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 34405 } }
I20260926 16:45:53.320399 5692 ts_tablet_manager.cc:1428] T c9e026a50abe4c5f899132e9dbcff7cf P 9847782936414dd38e658a1faf3d2234: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260926 16:45:53.320509 5802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:53.320231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.65:59171)
0926 16:45:53.320343 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:53.320402 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 16:45:53.321204 5802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:53.320972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.65:53743)
0926 16:45:53.321074 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:53.321118 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 16:45:53.413893 5803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:45:53.414129 5803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:45:53.414170 5803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:45:53.418183 5803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:45:53.418241 5803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:45:53.418331 5803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.66
I20260926 16:45:53.423557 5803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:45327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab
--principal=kudu/127.21.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.66:35041
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.66
--webserver_port=44039
--tserver_master_addrs=127.21.12.126:35725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.66
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:45:53.424698 5803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:45:53.425698 5803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:45:53.435236 5813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:45:53.435233 5810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:45:53.435236 5811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:45:53.435484 5803 server_base.cc:1034] running on GCE node
I20260926 16:45:53.436087 5803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:45:53.436745 5803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:45:53.437956 5803 hybrid_clock.cc:648] HybridClock initialized: now 1790441153437944 us; error 34 us; skew 500 ppm
Sep 26 16:45:53 dist-test-slave-n1hw krb5kdc[4473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441153, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:45:53.440735 5803 init.cc:377] Logged in from keytab as kudu/127.21.12.66@KRBTEST.COM (short username kudu)
I20260926 16:45:53.441552 5803 webserver.cc:460] Webserver started at http://127.21.12.66:44039/ using document root <none> and password file <none>
I20260926 16:45:53.442155 5803 fs_manager.cc:359] Metadata directory not provided
I20260926 16:45:53.442243 5803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:45:53.446058 5803 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260926 16:45:53.448225 5821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:45:53.449508 5803 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:45:53.449654 5803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-1/wal
uuid: "763e6b4243744af68be266b9d09c1aa7"
format_stamp: "Formatted at 2026-09-26 16:45:11 on dist-test-slave-n1hw"
server_key: "d9f8d6c1f50a439755e30ec80df2fe0d"
server_key_iv: "6b6111300153a74f160696fde1c3cc38"
server_key_version: "encryptionkey@0"
I20260926 16:45:53.450264 5803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:45:53.460956 5803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:45:53.463696 5803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:45:53.463907 5803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:45:53.464459 5803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:45:53.465467 5828 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 16:45:53.469347 5803 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 16:45:53.469424 5803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260926 16:45:53.469503 5803 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 16:45:53.471177 5803 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 16:45:53.471252 5803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260926 16:45:53.471483 5828 tablet_bootstrap.cc:492] T c9e026a50abe4c5f899132e9dbcff7cf P 763e6b4243744af68be266b9d09c1aa7: Bootstrap starting.
I20260926 16:45:53.481276 5803 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.66:35041
I20260926 16:45:53.481348 5936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.66:35041 every 8 connection(s)
I20260926 16:45:53.482262 5803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
I20260926 16:45:53.485119 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 5803
I20260926 16:45:53.485543 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.67:39145
--local_ip_for_outbound_sockets=127.21.12.67
--tserver_master_addrs=127.21.12.126:35725
--webserver_port=33883
--webserver_interface=127.21.12.67
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab
--principal=kudu/127.21.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:45327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:45:53.486002 5939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:53.484336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.66:59597)
0926 16:45:53.484871 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:53.485014 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":151,"threads_started":1}
W20260926 16:45:53.486704 5937 heartbeater.cc:644] Failed to heartbeat to 127.21.12.126:35725 (0 consecutive failures): Network error: Failed to ping master at 127.21.12.126:35725: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
I20260926 16:45:53.487720 5941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:53.487114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.66:49575)
0926 16:45:53.487569 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:53.487631 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":97,"threads_started":1}
I20260926 16:45:53.488317 5941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:53.488102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.66:59551)
0926 16:45:53.488212 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:53.488271 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 16:45:53.488528 5937 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 16:45:53.493391 5828 log.cc:824] T c9e026a50abe4c5f899132e9dbcff7cf P 763e6b4243744af68be266b9d09c1aa7: Log is configured to *not* fsync() on all Append() calls
I20260926 16:45:53.502504 5828 tablet_bootstrap.cc:492] T c9e026a50abe4c5f899132e9dbcff7cf P 763e6b4243744af68be266b9d09c1aa7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 16:45:53.502907 5828 tablet_bootstrap.cc:492] T c9e026a50abe4c5f899132e9dbcff7cf P 763e6b4243744af68be266b9d09c1aa7: Bootstrap complete.
I20260926 16:45:53.504370 5828 ts_tablet_manager.cc:1397] T c9e026a50abe4c5f899132e9dbcff7cf P 763e6b4243744af68be266b9d09c1aa7: Time spent bootstrapping tablet: real 0.033s user 0.025s sys 0.004s
I20260926 16:45:53.508052 5828 raft_consensus.cc:348] T c9e026a50abe4c5f899132e9dbcff7cf P 763e6b4243744af68be266b9d09c1aa7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20749a2d644876849e5f81fe7bd137" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 39145 } } peers { permanent_uuid: "763e6b4243744af68be266b9d09c1aa7" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 35041 } } peers { permanent_uuid: "9847782936414dd38e658a1faf3d2234" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 34405 } }
I20260926 16:45:53.508610 5828 raft_consensus.cc:729] T c9e026a50abe4c5f899132e9dbcff7cf P 763e6b4243744af68be266b9d09c1aa7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 763e6b4243744af68be266b9d09c1aa7, State: Initialized, Role: FOLLOWER
I20260926 16:45:53.509115 5828 consensus_queue.cc:260] T c9e026a50abe4c5f899132e9dbcff7cf P 763e6b4243744af68be266b9d09c1aa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20749a2d644876849e5f81fe7bd137" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 39145 } } peers { permanent_uuid: "763e6b4243744af68be266b9d09c1aa7" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 35041 } } peers { permanent_uuid: "9847782936414dd38e658a1faf3d2234" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 34405 } }
I20260926 16:45:53.510110 5828 ts_tablet_manager.cc:1428] T c9e026a50abe4c5f899132e9dbcff7cf P 763e6b4243744af68be266b9d09c1aa7: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260926 16:45:53.510476 5941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:53.510149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.66:42207)
0926 16:45:53.510342 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:53.510407 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260926 16:45:53.511292 5941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:53.511015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.66:53347)
0926 16:45:53.511176 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:53.511231 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260926 16:45:53.602772 5940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:45:53.603026 5940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:45:53.603068 5940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:45:53.607061 5940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:45:53.607116 5940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:45:53.607172 5940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.67
I20260926 16:45:53.612363 5940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:45327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab
--principal=kudu/127.21.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.67:39145
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.67
--webserver_port=33883
--tserver_master_addrs=127.21.12.126:35725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.67
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:45:53.613279 5940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:45:53.614248 5940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:45:53.623369 5948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:45:53.623377 5951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:45:53.623389 5949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:45:53.623653 5940 server_base.cc:1034] running on GCE node
I20260926 16:45:53.624070 5940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:45:53.624717 5940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:45:53.625886 5940 hybrid_clock.cc:648] HybridClock initialized: now 1790441153625864 us; error 32 us; skew 500 ppm
Sep 26 16:45:53 dist-test-slave-n1hw krb5kdc[4473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441153, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:45:53.628471 5940 init.cc:377] Logged in from keytab as kudu/127.21.12.67@KRBTEST.COM (short username kudu)
I20260926 16:45:53.629274 5940 webserver.cc:460] Webserver started at http://127.21.12.67:33883/ using document root <none> and password file <none>
I20260926 16:45:53.629885 5940 fs_manager.cc:359] Metadata directory not provided
I20260926 16:45:53.629976 5940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:45:53.633695 5940 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:45:53.635617 5958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:45:53.636839 5940 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 16:45:53.636973 5940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-2/wal
uuid: "ed20749a2d644876849e5f81fe7bd137"
format_stamp: "Formatted at 2026-09-26 16:45:11 on dist-test-slave-n1hw"
server_key: "f690a34952386bf16d25b145cbf3cd10"
server_key_iv: "4d36fb9263f7c3e3d48a57ee33948462"
server_key_version: "encryptionkey@0"
I20260926 16:45:53.637501 5940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:45:53.665220 5940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:45:53.668107 5940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:45:53.668321 5940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:45:53.668903 5940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:45:53.669713 5940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:45:53.669806 5940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:45:53.669896 5940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:45:53.669948 5940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:45:53.678630 5940 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.67:39145
I20260926 16:45:53.678656 6072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.67:39145 every 8 connection(s)
I20260926 16:45:53.679600 5940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
I20260926 16:45:53.683377 6075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:53.681675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.67:57273)
0926 16:45:53.682165 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:53.682268 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":115,"threads_started":1}
I20260926 16:45:53.683408 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 5940
W20260926 16:45:53.683858 6073 heartbeater.cc:644] Failed to heartbeat to 127.21.12.126:35725 (0 consecutive failures): Network error: Failed to ping master at 127.21.12.126:35725: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
I20260926 16:45:53.684437 6075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:53.684219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.67:57945)
0926 16:45:53.684331 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:53.684388 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 16:45:53.685051 6075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:53.684881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.67:44383)
0926 16:45:53.684967 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:53.685007 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 16:45:53.685258 6073 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 16:45:54.322695 6077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:54.322130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.65:41147)
0926 16:45:54.322480 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:54.322554 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":112,"threads_started":1}
I20260926 16:45:54.512390 6078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:54.512003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.66:47589)
0926 16:45:54.512230 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:54.512300 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":69,"threads_started":1}
I20260926 16:45:54.686453 6079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:54.685973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.67:38643)
0926 16:45:54.686288 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:54.686355 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":102,"threads_started":1}
I20260926 16:45:54.772941 6080 raft_consensus.cc:482] T c9e026a50abe4c5f899132e9dbcff7cf P 9847782936414dd38e658a1faf3d2234 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:45:54.773157 6080 raft_consensus.cc:504] T c9e026a50abe4c5f899132e9dbcff7cf P 9847782936414dd38e658a1faf3d2234 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20749a2d644876849e5f81fe7bd137" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 39145 } } peers { permanent_uuid: "763e6b4243744af68be266b9d09c1aa7" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 35041 } } peers { permanent_uuid: "9847782936414dd38e658a1faf3d2234" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 34405 } }
I20260926 16:45:54.774336 6080 leader_election.cc:288] T c9e026a50abe4c5f899132e9dbcff7cf P 9847782936414dd38e658a1faf3d2234 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ed20749a2d644876849e5f81fe7bd137 (127.21.12.67:39145), 763e6b4243744af68be266b9d09c1aa7 (127.21.12.66:35041)
Sep 26 16:45:54 dist-test-slave-n1hw krb5kdc[4473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441153, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for kudu/127.21.12.67@KRBTEST.COM
Sep 26 16:45:54 dist-test-slave-n1hw krb5kdc[4473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441153, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for kudu/127.21.12.66@KRBTEST.COM
I20260926 16:45:54.790971 6083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:54.774775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:40817 (local address 127.21.12.66:35041)
0926 16:45:54.775091 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0926 16:45:54.775108 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 16:45:54.775150 (+ 42us) server_negotiation.cc:407] Connection header received
0926 16:45:54.775429 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:45:54.775435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:45:54.775875 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:45:54.776211 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:45:54.777816 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:54.779396 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:54.781292 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:54.782053 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:54.782219 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:54.786052 (+ 3833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:45:54.786059 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:45:54.786075 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:45:54.786104 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:45:54.787932 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:45:54.788466 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:45:54.788474 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:45:54.788483 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:45:54.788739 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:45:54.789061 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:45:54.789066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:45:54.789070 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:45:54.789534 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:45:54.789692 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 16:45:54.790471 (+ 779us) server_negotiation.cc:299] Negotiation successful
0926 16:45:54.790769 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":61,"threads_started":1}
I20260926 16:45:54.790967 6082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:54.774677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:53415 (local address 127.21.12.67:39145)
0926 16:45:54.775091 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0926 16:45:54.775112 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 16:45:54.775150 (+ 38us) server_negotiation.cc:407] Connection header received
0926 16:45:54.776012 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:45:54.776022 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:45:54.776455 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:45:54.776741 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:45:54.777816 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:54.779396 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:54.781296 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:54.782053 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:54.782226 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:54.785599 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:45:54.785604 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:45:54.785617 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:45:54.785645 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:45:54.787782 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:45:54.788516 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:45:54.788521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:45:54.788527 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:45:54.788648 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:45:54.789098 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:45:54.789101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:45:54.789102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:45:54.789543 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:45:54.789814 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0926 16:45:54.790471 (+ 657us) server_negotiation.cc:299] Negotiation successful
0926 16:45:54.790757 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":93,"threads_started":1}
I20260926 16:45:54.790978 6077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:54.774526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.67:39145 (local address 127.21.12.65:53415)
0926 16:45:54.774731 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:54.774748 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:45:54.774850 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:45:54.776728 (+ 1878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:45:54.776735 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:45:54.776886 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:45:54.777582 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:54.777602 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:54.779590 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:54.779594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:54.781135 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:54.781146 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:54.782181 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:54.782185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:54.782447 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:54.783199 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:45:54.783220 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:45:54.785340 (+ 2120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:45:54.787914 (+ 2574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:45:54.787921 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:45:54.787935 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:45:54.788367 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:45:54.788788 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:45:54.788793 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:45:54.788799 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:45:54.788982 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:45:54.789681 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:45:54.789691 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:45:54.790075 (+ 384us) client_negotiation.cc:769] Sending connection context
0926 16:45:54.790331 (+ 256us) client_negotiation.cc:240] Negotiation successful
0926 16:45:54.790802 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":553,"spinlock_wait_cycles":35584}
I20260926 16:45:54.790977 6081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:54.774560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:35041 (local address 127.21.12.65:40817)
0926 16:45:54.774896 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:54.774909 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 16:45:54.775008 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:45:54.776320 (+ 1312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:45:54.776325 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:45:54.776776 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:45:54.777582 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:54.777602 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:54.779590 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:54.779594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:54.781136 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:54.781146 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:54.782181 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:54.782185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:54.782444 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:54.783215 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:45:54.783228 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:45:54.785881 (+ 2653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:45:54.788046 (+ 2165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:45:54.788049 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:45:54.788050 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:45:54.788351 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:45:54.788840 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:45:54.788842 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:45:54.788843 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:45:54.788961 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:45:54.789691 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:45:54.789694 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:45:54.790075 (+ 381us) client_negotiation.cc:769] Sending connection context
0926 16:45:54.790331 (+ 256us) client_negotiation.cc:240] Negotiation successful
0926 16:45:54.790746 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":349,"spinlock_wait_cycles":34816,"thread_start_us":70,"threads_started":1}
I20260926 16:45:54.792934 5891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e026a50abe4c5f899132e9dbcff7cf" candidate_uuid: "9847782936414dd38e658a1faf3d2234" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "763e6b4243744af68be266b9d09c1aa7" is_pre_election: true
I20260926 16:45:54.793265 5891 raft_consensus.cc:2454] T c9e026a50abe4c5f899132e9dbcff7cf P 763e6b4243744af68be266b9d09c1aa7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9847782936414dd38e658a1faf3d2234 in term 1.
W20260926 16:45:54.793602 5689 leader_election.cc:341] T c9e026a50abe4c5f899132e9dbcff7cf P 9847782936414dd38e658a1faf3d2234 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ed20749a2d644876849e5f81fe7bd137 (127.21.12.67:39145): Not found: Tablet not found: c9e026a50abe4c5f899132e9dbcff7cf
I20260926 16:45:54.794049 5688 leader_election.cc:302] T c9e026a50abe4c5f899132e9dbcff7cf P 9847782936414dd38e658a1faf3d2234 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 763e6b4243744af68be266b9d09c1aa7, 9847782936414dd38e658a1faf3d2234; no voters: ed20749a2d644876849e5f81fe7bd137
I20260926 16:45:54.794354 6080 raft_consensus.cc:2781] T c9e026a50abe4c5f899132e9dbcff7cf P 9847782936414dd38e658a1faf3d2234 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 16:45:54.794437 6080 raft_consensus.cc:482] T c9e026a50abe4c5f899132e9dbcff7cf P 9847782936414dd38e658a1faf3d2234 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:45:54.794482 6080 raft_consensus.cc:3037] T c9e026a50abe4c5f899132e9dbcff7cf P 9847782936414dd38e658a1faf3d2234 [term 1 FOLLOWER]: Advancing to term 2
I20260926 16:45:54.796507 6080 raft_consensus.cc:504] T c9e026a50abe4c5f899132e9dbcff7cf P 9847782936414dd38e658a1faf3d2234 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20749a2d644876849e5f81fe7bd137" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 39145 } } peers { permanent_uuid: "763e6b4243744af68be266b9d09c1aa7" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 35041 } } peers { permanent_uuid: "9847782936414dd38e658a1faf3d2234" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 34405 } }
I20260926 16:45:54.796903 6080 leader_election.cc:288] T c9e026a50abe4c5f899132e9dbcff7cf P 9847782936414dd38e658a1faf3d2234 [CANDIDATE]: Term 2 election: Requested vote from peers ed20749a2d644876849e5f81fe7bd137 (127.21.12.67:39145), 763e6b4243744af68be266b9d09c1aa7 (127.21.12.66:35041)
I20260926 16:45:54.797295 5891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e026a50abe4c5f899132e9dbcff7cf" candidate_uuid: "9847782936414dd38e658a1faf3d2234" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "763e6b4243744af68be266b9d09c1aa7"
I20260926 16:45:54.797439 5891 raft_consensus.cc:3037] T c9e026a50abe4c5f899132e9dbcff7cf P 763e6b4243744af68be266b9d09c1aa7 [term 1 FOLLOWER]: Advancing to term 2
W20260926 16:45:54.797848 5689 leader_election.cc:341] T c9e026a50abe4c5f899132e9dbcff7cf P 9847782936414dd38e658a1faf3d2234 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer ed20749a2d644876849e5f81fe7bd137 (127.21.12.67:39145): Not found: Tablet not found: c9e026a50abe4c5f899132e9dbcff7cf
I20260926 16:45:54.799533 5891 raft_consensus.cc:2454] T c9e026a50abe4c5f899132e9dbcff7cf P 763e6b4243744af68be266b9d09c1aa7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9847782936414dd38e658a1faf3d2234 in term 2.
I20260926 16:45:54.799873 5688 leader_election.cc:302] T c9e026a50abe4c5f899132e9dbcff7cf P 9847782936414dd38e658a1faf3d2234 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 763e6b4243744af68be266b9d09c1aa7, 9847782936414dd38e658a1faf3d2234; no voters: ed20749a2d644876849e5f81fe7bd137
I20260926 16:45:54.800051 6080 raft_consensus.cc:2781] T c9e026a50abe4c5f899132e9dbcff7cf P 9847782936414dd38e658a1faf3d2234 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 16:45:54.800283 6080 raft_consensus.cc:686] T c9e026a50abe4c5f899132e9dbcff7cf P 9847782936414dd38e658a1faf3d2234 [term 2 LEADER]: Becoming Leader. State: Replica: 9847782936414dd38e658a1faf3d2234, State: Running, Role: LEADER
I20260926 16:45:54.800634 6080 consensus_queue.cc:237] T c9e026a50abe4c5f899132e9dbcff7cf P 9847782936414dd38e658a1faf3d2234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20749a2d644876849e5f81fe7bd137" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 39145 } } peers { permanent_uuid: "763e6b4243744af68be266b9d09c1aa7" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 35041 } } peers { permanent_uuid: "9847782936414dd38e658a1faf3d2234" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 34405 } }
I20260926 16:45:54.801049 6077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:54.800729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.65:59501)
0926 16:45:54.800854 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:54.800929 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 16:45:55.229398 5689 consensus_peers.cc:487] T c9e026a50abe4c5f899132e9dbcff7cf P 9847782936414dd38e658a1faf3d2234 -> Peer ed20749a2d644876849e5f81fe7bd137 (127.21.12.67:39145): Couldn't send request to peer ed20749a2d644876849e5f81fe7bd137. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c9e026a50abe4c5f899132e9dbcff7cf. This is attempt 1: this message will repeat every 5th retry.
I20260926 16:45:55.281170 5891 raft_consensus.cc:1261] T c9e026a50abe4c5f899132e9dbcff7cf P 763e6b4243744af68be266b9d09c1aa7 [term 2 FOLLOWER]: Refusing update from remote peer 9847782936414dd38e658a1faf3d2234: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260926 16:45:55.282016 6080 consensus_queue.cc:1035] T c9e026a50abe4c5f899132e9dbcff7cf P 9847782936414dd38e658a1faf3d2234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "763e6b4243744af68be266b9d09c1aa7" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 35041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260926 16:45:55.284113 6087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:55.283688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.66:42691)
0926 16:45:55.283935 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:55.283999 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":66,"threads_started":1}
I20260926 16:45:55.286314 6077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:55.285980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.65:40437)
0926 16:45:55.286178 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:55.286235 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260926 16:45:55.286625 6087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:55.286393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.66:35175)
0926 16:45:55.286505 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:55.286561 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 16:45:55.687726 6091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:55.687286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.67:60585)
0926 16:45:55.687559 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:55.687616 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":89,"threads_started":1}
I20260926 16:45:55.773253 6092 ts_tablet_manager.cc:927] T c9e026a50abe4c5f899132e9dbcff7cf P ed20749a2d644876849e5f81fe7bd137: Initiating tablet copy from peer 9847782936414dd38e658a1faf3d2234 (127.21.12.65:34405)
I20260926 16:45:55.773990 6092 tablet_copy_client.cc:316] T c9e026a50abe4c5f899132e9dbcff7cf P ed20749a2d644876849e5f81fe7bd137: tablet copy: Beginning tablet copy session from remote peer at address 127.21.12.65:34405
Sep 26 16:45:55 dist-test-slave-n1hw krb5kdc[4473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441153, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.67@KRBTEST.COM for kudu/127.21.12.65@KRBTEST.COM
I20260926 16:45:55.784260 6091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:55.774545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:34405 (local address 127.21.12.67:57391)
0926 16:45:55.774685 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:55.774715 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 16:45:55.774832 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:45:55.775518 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:45:55.775523 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:45:55.775913 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:45:55.776031 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:55.776036 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:55.776999 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:55.777002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:55.777898 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:55.777908 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:55.778474 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:55.778477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:55.778557 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:55.779199 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:45:55.779212 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:45:55.780622 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:45:55.782608 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:45:55.782616 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:45:55.782621 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:45:55.782877 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:45:55.783190 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:45:55.783193 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:45:55.783194 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:45:55.783277 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:45:55.783759 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:45:55.783765 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:45:55.783875 (+ 110us) client_negotiation.cc:769] Sending connection context
0926 16:45:55.784000 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 16:45:55.784134 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 16:45:55.784349 6094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:55.774689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:57391 (local address 127.21.12.65:34405)
0926 16:45:55.774950 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0926 16:45:55.774959 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:45:55.774989 (+ 30us) server_negotiation.cc:407] Connection header received
0926 16:45:55.775033 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:45:55.775037 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:45:55.775312 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:45:55.775495 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:45:55.776139 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:55.776883 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:55.778007 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:55.778365 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:55.778452 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:55.780787 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:45:55.780797 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:45:55.780809 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:45:55.780831 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:45:55.782495 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:45:55.782993 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:45:55.782998 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:45:55.782999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:45:55.783081 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:45:55.783367 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:45:55.783370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:45:55.783371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:45:55.783618 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:45:55.783752 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 16:45:55.784085 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 16:45:55.784223 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":104,"threads_started":1}
I20260926 16:45:55.785435 5774 tablet_copy_service.cc:140] P 9847782936414dd38e658a1faf3d2234: Received BeginTabletCopySession request for tablet c9e026a50abe4c5f899132e9dbcff7cf from peer ed20749a2d644876849e5f81fe7bd137 ({username='kudu', principal='kudu/127.21.12.67@KRBTEST.COM'} at 127.21.12.67:57391)
I20260926 16:45:55.785633 5774 tablet_copy_service.cc:161] P 9847782936414dd38e658a1faf3d2234: Beginning new tablet copy session on tablet c9e026a50abe4c5f899132e9dbcff7cf from peer ed20749a2d644876849e5f81fe7bd137 at {username='kudu', principal='kudu/127.21.12.67@KRBTEST.COM'} at 127.21.12.67:57391: session id = ed20749a2d644876849e5f81fe7bd137-c9e026a50abe4c5f899132e9dbcff7cf
I20260926 16:45:55.787187 5774 tablet_copy_source_session.cc:205] T c9e026a50abe4c5f899132e9dbcff7cf P 9847782936414dd38e658a1faf3d2234: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 16:45:55.791287 6092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9e026a50abe4c5f899132e9dbcff7cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:45:55.797575 6092 tablet_copy_client.cc:791] T c9e026a50abe4c5f899132e9dbcff7cf P ed20749a2d644876849e5f81fe7bd137: tablet copy: Starting download of 0 data blocks...
I20260926 16:45:55.797817 6092 tablet_copy_client.cc:655] T c9e026a50abe4c5f899132e9dbcff7cf P ed20749a2d644876849e5f81fe7bd137: tablet copy: Starting download of 1 WAL segments...
I20260926 16:45:55.799798 6092 tablet_copy_client.cc:531] T c9e026a50abe4c5f899132e9dbcff7cf P ed20749a2d644876849e5f81fe7bd137: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 16:45:55.801889 6092 tablet_bootstrap.cc:492] T c9e026a50abe4c5f899132e9dbcff7cf P ed20749a2d644876849e5f81fe7bd137: Bootstrap starting.
I20260926 16:45:55.807185 6092 log.cc:824] T c9e026a50abe4c5f899132e9dbcff7cf P ed20749a2d644876849e5f81fe7bd137: Log is configured to *not* fsync() on all Append() calls
I20260926 16:45:55.817456 6092 tablet_bootstrap.cc:492] T c9e026a50abe4c5f899132e9dbcff7cf P ed20749a2d644876849e5f81fe7bd137: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 16:45:55.817981 6092 tablet_bootstrap.cc:492] T c9e026a50abe4c5f899132e9dbcff7cf P ed20749a2d644876849e5f81fe7bd137: Bootstrap complete.
I20260926 16:45:55.818351 6092 ts_tablet_manager.cc:1397] T c9e026a50abe4c5f899132e9dbcff7cf P ed20749a2d644876849e5f81fe7bd137: Time spent bootstrapping tablet: real 0.017s user 0.016s sys 0.001s
I20260926 16:45:55.822507 6092 raft_consensus.cc:348] T c9e026a50abe4c5f899132e9dbcff7cf P ed20749a2d644876849e5f81fe7bd137 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20749a2d644876849e5f81fe7bd137" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 39145 } } peers { permanent_uuid: "763e6b4243744af68be266b9d09c1aa7" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 35041 } } peers { permanent_uuid: "9847782936414dd38e658a1faf3d2234" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 34405 } }
I20260926 16:45:55.822873 6092 raft_consensus.cc:729] T c9e026a50abe4c5f899132e9dbcff7cf P ed20749a2d644876849e5f81fe7bd137 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed20749a2d644876849e5f81fe7bd137, State: Initialized, Role: FOLLOWER
I20260926 16:45:55.823423 6092 consensus_queue.cc:260] T c9e026a50abe4c5f899132e9dbcff7cf P ed20749a2d644876849e5f81fe7bd137 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20749a2d644876849e5f81fe7bd137" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 39145 } } peers { permanent_uuid: "763e6b4243744af68be266b9d09c1aa7" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 35041 } } peers { permanent_uuid: "9847782936414dd38e658a1faf3d2234" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 34405 } }
I20260926 16:45:55.824589 6091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:55.824343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.67:36883)
0926 16:45:55.824467 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:55.824525 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 16:45:55.824609 6092 ts_tablet_manager.cc:1428] T c9e026a50abe4c5f899132e9dbcff7cf P ed20749a2d644876849e5f81fe7bd137: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 16:45:55.825362 5774 tablet_copy_service.cc:342] P 9847782936414dd38e658a1faf3d2234: Request end of tablet copy session ed20749a2d644876849e5f81fe7bd137-c9e026a50abe4c5f899132e9dbcff7cf received from {username='kudu', principal='kudu/127.21.12.67@KRBTEST.COM'} at 127.21.12.67:57391
I20260926 16:45:55.825448 6091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:55.825121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.67:41091)
0926 16:45:55.825285 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:55.825355 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 16:45:55.825488 5774 tablet_copy_service.cc:433] P 9847782936414dd38e658a1faf3d2234: ending tablet copy session ed20749a2d644876849e5f81fe7bd137-c9e026a50abe4c5f899132e9dbcff7cf on tablet c9e026a50abe4c5f899132e9dbcff7cf with peer ed20749a2d644876849e5f81fe7bd137
I20260926 16:45:56.278388 6099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:56.277933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.65:33541)
0926 16:45:56.278169 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:56.278228 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":60,"threads_started":1}
I20260926 16:45:56.278407 6027 raft_consensus.cc:1203] T c9e026a50abe4c5f899132e9dbcff7cf P ed20749a2d644876849e5f81fe7bd137 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260926 16:45:56.279331 6091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:56.279085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.67:56151)
0926 16:45:56.279201 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:56.279255 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 16:45:56.287495 6100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:56.287182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.66:39159)
0926 16:45:56.287372 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:56.287420 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":55,"threads_started":1}
I20260926 16:45:57.279547 6101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:57.279143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.65:45469)
0926 16:45:57.279398 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:57.279455 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":73,"threads_started":1}
I20260926 16:45:57.280340 6102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:57.279956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.67:52769)
0926 16:45:57.280209 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:57.280259 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":62,"threads_started":1}
I20260926 16:45:57.288579 6103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:57.288148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.66:53565)
0926 16:45:57.288452 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:57.288502 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":67,"threads_started":1}
I20260926 16:45:58.280720 6104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:58.280274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.65:45755)
0926 16:45:58.280527 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:58.280579 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":73,"threads_started":1}
I20260926 16:45:58.281389 6105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:58.281024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.67:59859)
0926 16:45:58.281263 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:58.281312 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":75,"threads_started":1}
I20260926 16:45:58.289674 6106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:58.289274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.66:53969)
0926 16:45:58.289546 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:58.289595 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":88,"threads_started":1}
I20260926 16:45:59.282114 6107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:59.281690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.65:59037)
0926 16:45:59.281966 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:59.282022 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":119,"threads_started":1}
I20260926 16:45:59.282649 6108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:59.282287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.67:55557)
0926 16:45:59.282516 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:59.282576 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":99,"threads_started":1}
I20260926 16:45:59.282859 6107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:59.282624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.65:50417)
0926 16:45:59.282741 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:59.282792 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 16:45:59.290817 6109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:59.290444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.66:44429)
0926 16:45:59.290670 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:59.290721 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":81,"threads_started":1}
I20260926 16:46:00.284008 6111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:00.283572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.65:59825)
0926 16:46:00.283864 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:00.283921 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":93,"threads_started":1}
I20260926 16:46:00.284010 6110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:00.283509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.67:45483)
0926 16:46:00.283802 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:00.283887 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":118,"threads_started":1}
I20260926 16:46:00.292016 6112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:00.291619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.66:59137)
0926 16:46:00.291867 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:00.291922 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":68,"threads_started":1}
I20260926 16:46:01.285158 6113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:01.284772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.67:60147)
0926 16:46:01.285018 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:01.285068 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":89,"threads_started":1}
I20260926 16:46:01.285259 6115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:01.284869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.65:39685)
0926 16:46:01.285128 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:01.285179 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":107,"threads_started":1}
I20260926 16:46:01.293169 6116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:01.292734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.66:45375)
0926 16:46:01.293049 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:01.293094 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":62,"threads_started":1}
I20260926 16:46:02.286444 6118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:02.285971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.65:35455)
0926 16:46:02.286228 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:02.286300 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":70,"threads_started":1}
I20260926 16:46:02.286535 6119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:02.285978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.67:45351)
0926 16:46:02.286303 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:02.286400 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":94,"threads_started":1}
I20260926 16:46:02.294224 6120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:02.293853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.66:48045)
0926 16:46:02.294092 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:02.294142 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":86,"threads_started":1}
I20260926 16:46:03.287639 6123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:03.287207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.67:38367)
0926 16:46:03.287492 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:03.287548 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":88,"threads_started":1}
I20260926 16:46:03.287649 6124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:03.287237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.65:41311)
0926 16:46:03.287516 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:03.287568 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":95,"threads_started":1}
I20260926 16:46:03.295383 6125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:03.294975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.66:39705)
0926 16:46:03.295225 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:03.295296 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":98,"threads_started":1}
I20260926 16:46:04.288879 6127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:04.288446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.65:56505)
0926 16:46:04.288734 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:04.288787 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":111,"threads_started":1}
I20260926 16:46:04.288957 6126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:04.288474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.67:40917)
0926 16:46:04.288807 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:04.288857 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":90,"threads_started":1}
I20260926 16:46:04.296525 6128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:04.296161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.66:35283)
0926 16:46:04.296393 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:04.296446 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":73,"threads_started":1}
I20260926 16:46:05.290156 6130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:05.289724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.65:39881)
0926 16:46:05.290000 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:05.290061 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":105,"threads_started":1}
I20260926 16:46:05.290239 6131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:05.289724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.67:33917)
0926 16:46:05.290082 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:05.290132 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":67,"threads_started":1}
I20260926 16:46:05.297621 6132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:05.297210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.66:56437)
0926 16:46:05.297460 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:05.297521 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":71,"threads_started":1}
I20260926 16:46:06.291294 6133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:06.290894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.65:33329)
0926 16:46:06.291153 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:06.291205 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":101,"threads_started":1}
I20260926 16:46:06.291405 6134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:06.290894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.67:35685)
0926 16:46:06.291238 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:06.291288 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":95,"threads_started":1}
I20260926 16:46:06.298774 6135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:06.298348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.66:36895)
0926 16:46:06.298641 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:06.298693 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":137,"threads_started":1}
I20260926 16:46:07.292346 6136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:07.291959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.65:47197)
0926 16:46:07.292200 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:07.292250 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":102,"threads_started":1}
I20260926 16:46:07.292433 6137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:07.292072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.67:41967)
0926 16:46:07.292302 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:07.292352 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":85,"threads_started":1}
I20260926 16:46:07.299845 6138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:07.299482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.66:38745)
0926 16:46:07.299706 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:07.299759 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":59,"threads_started":1}
I20260926 16:46:08.293668 6140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:08.293205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.67:47195)
0926 16:46:08.293503 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:08.293563 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":85,"threads_started":1}
I20260926 16:46:08.293699 6141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:08.293205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.65:51997)
0926 16:46:08.293570 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:08.293620 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":89,"threads_started":1}
I20260926 16:46:08.301079 6142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:08.300601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:35725 (local address 127.21.12.66:53933)
0926 16:46:08.300873 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:08.300955 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.126:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":84,"threads_started":1}
I20260926 16:46:08.686836 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 5630
I20260926 16:46:08.692863 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 5803
I20260926 16:46:08.698124 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 5940
2026-09-26T16:46:08Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59432 ms)
[----------] 30 tests from SecurityITest (243755 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:46:08 dist-test-slave-n1hw krb5kdc[6145](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:46:08 dist-test-slave-n1hw krb5kdc[6145](info): set up 2 sockets
Sep 26 16:46:08 dist-test-slave-n1hw krb5kdc[6145](info): commencing operation
krb5kdc: starting...
W20260926 16:46:09.739667 21553 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Sep 26 16:46:09 dist-test-slave-n1hw krb5kdc[6145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:46:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:46:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
I20260926 16:46:09.862185 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:45555
--webserver_interface=127.21.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:40079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:45555
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:46:09.970422 6158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:09.970716 6158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:09.970794 6158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:09.974939 6158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:46:09.975018 6158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:09.975056 6158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:09.975114 6158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:46:09.975154 6158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:46:09.980383 6158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:40079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:45555
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:45555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:46:09.981487 6158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:09.982596 6158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:09.991261 6164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:09.991289 6163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:09.991353 6158 server_base.cc:1034] running on GCE node
W20260926 16:46:09.991289 6166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:09.992166 6158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:09.993501 6158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:09.994711 6158 hybrid_clock.cc:648] HybridClock initialized: now 1790441169994674 us; error 47 us; skew 500 ppm
Sep 26 16:46:09 dist-test-slave-n1hw krb5kdc[6145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441169, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:46:09.997715 6158 init.cc:377] Logged in from keytab as kudu/127.21.12.126@KRBTEST.COM (short username kudu)
I20260926 16:46:09.998708 6158 webserver.cc:460] Webserver started at http://127.21.12.126:39709/ using document root <none> and password file <none>
I20260926 16:46:09.999305 6158 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:09.999369 6158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:09.999672 6158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:46:10.001717 6158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "1e49f63a3db743b7a234d1cfd0b9359e"
format_stamp: "Formatted at 2026-09-26 16:46:09 on dist-test-slave-n1hw"
server_key: "12069d80ad2e7c916141566c810b45fb"
server_key_iv: "ad0fbf5b772172cbf358170ed3318504"
server_key_version: "encryptionkey@0"
I20260926 16:46:10.002369 6158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "1e49f63a3db743b7a234d1cfd0b9359e"
format_stamp: "Formatted at 2026-09-26 16:46:09 on dist-test-slave-n1hw"
server_key: "12069d80ad2e7c916141566c810b45fb"
server_key_iv: "ad0fbf5b772172cbf358170ed3318504"
server_key_version: "encryptionkey@0"
I20260926 16:46:10.006289 6158 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:46:10.008831 6173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:10.010200 6158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:46:10.010329 6158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "1e49f63a3db743b7a234d1cfd0b9359e"
format_stamp: "Formatted at 2026-09-26 16:46:09 on dist-test-slave-n1hw"
server_key: "12069d80ad2e7c916141566c810b45fb"
server_key_iv: "ad0fbf5b772172cbf358170ed3318504"
server_key_version: "encryptionkey@0"
I20260926 16:46:10.010442 6158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:10.022464 6158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:10.025491 6158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:10.025698 6158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:10.033015 6158 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.126:45555
I20260926 16:46:10.033046 6225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.126:45555 every 8 connection(s)
I20260926 16:46:10.034044 6158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:46:10.036813 6226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:10.038039 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 6158
I20260926 16:46:10.038182 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:46:10.038467 21553 external_mini_cluster.cc:1428] Setting key 382cb7aa870456bb4b6b7c46ab216fd1
I20260926 16:46:10.042840 6226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e49f63a3db743b7a234d1cfd0b9359e: Bootstrap starting.
I20260926 16:46:10.045059 6226 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e49f63a3db743b7a234d1cfd0b9359e: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:10.045966 6226 log.cc:824] T 00000000000000000000000000000000 P 1e49f63a3db743b7a234d1cfd0b9359e: Log is configured to *not* fsync() on all Append() calls
Sep 26 16:46:10 dist-test-slave-n1hw krb5kdc[6145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:46:10.048318 6226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e49f63a3db743b7a234d1cfd0b9359e: No bootstrap required, opened a new log
I20260926 16:46:10.052609 6226 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e49f63a3db743b7a234d1cfd0b9359e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e49f63a3db743b7a234d1cfd0b9359e" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 45555 } }
I20260926 16:46:10.052821 6226 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e49f63a3db743b7a234d1cfd0b9359e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:46:10.052855 6226 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e49f63a3db743b7a234d1cfd0b9359e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e49f63a3db743b7a234d1cfd0b9359e, State: Initialized, Role: FOLLOWER
I20260926 16:46:10.053020 6229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:10.039884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49938 (local address 127.21.12.126:45555)
0926 16:46:10.040334 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:10.040349 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:10.040386 (+ 37us) server_negotiation.cc:407] Connection header received
0926 16:46:10.041027 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:10.041044 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:10.041370 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:10.041737 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:10.042595 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:10.043610 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:10.044485 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:10.044964 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:10.045145 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:10.048139 (+ 2994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:10.048144 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:10.048157 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:10.048185 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:10.050001 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:10.050572 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:10.050577 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:10.050581 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:10.050692 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:10.050994 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:10.050997 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:10.050998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:10.051388 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:10.051558 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:10.051847 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 16:46:10.052166 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":122,"threads_started":1}
I20260926 16:46:10.053397 6226 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e49f63a3db743b7a234d1cfd0b9359e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e49f63a3db743b7a234d1cfd0b9359e" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 45555 } }
I20260926 16:46:10.053555 6226 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e49f63a3db743b7a234d1cfd0b9359e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:46:10.053644 6226 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e49f63a3db743b7a234d1cfd0b9359e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:46:10.053746 6226 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e49f63a3db743b7a234d1cfd0b9359e [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:10.054873 6226 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e49f63a3db743b7a234d1cfd0b9359e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e49f63a3db743b7a234d1cfd0b9359e" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 45555 } }
I20260926 16:46:10.055227 6226 leader_election.cc:302] T 00000000000000000000000000000000 P 1e49f63a3db743b7a234d1cfd0b9359e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e49f63a3db743b7a234d1cfd0b9359e; no voters:
I20260926 16:46:10.055616 6226 leader_election.cc:288] T 00000000000000000000000000000000 P 1e49f63a3db743b7a234d1cfd0b9359e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:46:10.055686 6231 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e49f63a3db743b7a234d1cfd0b9359e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:46:10.055990 6231 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e49f63a3db743b7a234d1cfd0b9359e [term 1 LEADER]: Becoming Leader. State: Replica: 1e49f63a3db743b7a234d1cfd0b9359e, State: Running, Role: LEADER
I20260926 16:46:10.056401 6231 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e49f63a3db743b7a234d1cfd0b9359e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e49f63a3db743b7a234d1cfd0b9359e" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 45555 } }
I20260926 16:46:10.056864 6226 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e49f63a3db743b7a234d1cfd0b9359e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:46:10.058077 6233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e49f63a3db743b7a234d1cfd0b9359e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e49f63a3db743b7a234d1cfd0b9359e. Latest consensus state: current_term: 1 leader_uuid: "1e49f63a3db743b7a234d1cfd0b9359e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e49f63a3db743b7a234d1cfd0b9359e" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 45555 } } }
I20260926 16:46:10.058207 6233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e49f63a3db743b7a234d1cfd0b9359e [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:10.058614 6232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e49f63a3db743b7a234d1cfd0b9359e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e49f63a3db743b7a234d1cfd0b9359e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e49f63a3db743b7a234d1cfd0b9359e" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 45555 } } }
I20260926 16:46:10.058732 6232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e49f63a3db743b7a234d1cfd0b9359e [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:10.059028 6240 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:46:10.062749 6240 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:46:10.068202 6240 catalog_manager.cc:1347] Generated new cluster ID: 7a5daba6d0dc41d9b05698c0e9c3d413
I20260926 16:46:10.068279 6240 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:46:10.083263 6240 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:46:10.084348 6240 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:46:10.090507 6240 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e49f63a3db743b7a234d1cfd0b9359e: Generated new TSK 0
I20260926 16:46:10.091377 6240 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab.
Entry for principal kudu/127.21.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab.
I20260926 16:46:10.131253 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.65:0
--local_ip_for_outbound_sockets=127.21.12.65
--webserver_interface=127.21.12.65
--webserver_port=0
--tserver_master_addrs=127.21.12.126:45555
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab
--principal=kudu/127.21.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:40079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:46:10.257509 6252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:10.257885 6252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:10.257968 6252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:10.262162 6252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:10.262235 6252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:10.262322 6252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.65
I20260926 16:46:10.267860 6252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:40079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab
--principal=kudu/127.21.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.65
--webserver_port=0
--tserver_master_addrs=127.21.12.126:45555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.65
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:46:10.268903 6252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:10.270000 6252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:10.279596 6261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:10.279613 6257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:10.279755 6252 server_base.cc:1034] running on GCE node
W20260926 16:46:10.279940 6259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:10.280314 6252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:10.280941 6252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:10.282132 6252 hybrid_clock.cc:648] HybridClock initialized: now 1790441170282120 us; error 30 us; skew 500 ppm
Sep 26 16:46:10 dist-test-slave-n1hw krb5kdc[6145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441170, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:46:10.284889 6252 init.cc:377] Logged in from keytab as kudu/127.21.12.65@KRBTEST.COM (short username kudu)
I20260926 16:46:10.285701 6252 webserver.cc:460] Webserver started at http://127.21.12.65:39819/ using document root <none> and password file <none>
I20260926 16:46:10.286274 6252 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:10.286365 6252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:10.286590 6252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:46:10.288622 6252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-0/data/instance:
uuid: "69e0d7aecc6148afb31c600b63b4c35d"
format_stamp: "Formatted at 2026-09-26 16:46:10 on dist-test-slave-n1hw"
server_key: "ef91720b95e9a10ce1e479e0edeaab64"
server_key_iv: "2d50f76adb8b64895ed321bd15e3492c"
server_key_version: "encryptionkey@0"
I20260926 16:46:10.289186 6252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance:
uuid: "69e0d7aecc6148afb31c600b63b4c35d"
format_stamp: "Formatted at 2026-09-26 16:46:10 on dist-test-slave-n1hw"
server_key: "ef91720b95e9a10ce1e479e0edeaab64"
server_key_iv: "2d50f76adb8b64895ed321bd15e3492c"
server_key_version: "encryptionkey@0"
I20260926 16:46:10.292793 6252 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260926 16:46:10.295229 6268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:10.296451 6252 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 16:46:10.296586 6252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-0/wal
uuid: "69e0d7aecc6148afb31c600b63b4c35d"
format_stamp: "Formatted at 2026-09-26 16:46:10 on dist-test-slave-n1hw"
server_key: "ef91720b95e9a10ce1e479e0edeaab64"
server_key_iv: "2d50f76adb8b64895ed321bd15e3492c"
server_key_version: "encryptionkey@0"
I20260926 16:46:10.296698 6252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:10.321699 6252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:10.325248 6252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:10.325460 6252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:10.326136 6252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:46:10.326984 6252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:46:10.327064 6252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:10.327157 6252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:46:10.327207 6252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:10.336084 6252 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.65:34247
I20260926 16:46:10.336102 6381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.65:34247 every 8 connection(s)
I20260926 16:46:10.336999 6252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
I20260926 16:46:10.338634 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 6252
I20260926 16:46:10.338766 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance
I20260926 16:46:10.339080 21553 external_mini_cluster.cc:1428] Setting key c5bb5821bfc38b26cbce53cac7c0814e
Sep 26 16:46:10 dist-test-slave-n1hw krb5kdc[6145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441170, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.12.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.66@KRBTEST.COM" created.
I20260926 16:46:10.352510 6229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:10.339025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:42083 (local address 127.21.12.126:45555)
0926 16:46:10.339182 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:10.339186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:10.340178 (+ 992us) server_negotiation.cc:407] Connection header received
0926 16:46:10.341590 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:10.341593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:10.341646 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:10.341724 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:10.343397 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:10.343936 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:10.345155 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:10.345394 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:10.345477 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:10.348349 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:10.348353 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:10.348355 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:10.348378 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:10.350089 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:10.350725 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:10.350728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:10.350730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:10.350814 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:10.351225 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:10.351228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:10.351230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:10.351455 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:10.351566 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:10.352099 (+ 533us) server_negotiation.cc:299] Negotiation successful
0926 16:46:10.352252 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 16:46:10.353338 6384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:10.339540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:45555 (local address 127.21.12.65:42083)
0926 16:46:10.340030 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:10.340072 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:10.341381 (+ 1309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:10.341945 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:10.341957 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:10.342504 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:46:10.343228 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:10.343251 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:10.344075 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:10.344078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:10.345024 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:10.345037 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:10.345558 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:10.345565 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:10.345857 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:10.346494 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:46:10.346521 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:46:10.348169 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:46:10.350210 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:10.350216 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:10.350232 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:10.350619 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:10.350952 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:10.350956 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:10.350958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:10.351118 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:10.351587 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:46:10.351592 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:46:10.351831 (+ 239us) client_negotiation.cc:769] Sending connection context
0926 16:46:10.352017 (+ 186us) client_negotiation.cc:240] Negotiation successful
0926 16:46:10.352275 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":109,"threads_started":1}
I20260926 16:46:10.354759 6382 heartbeater.cc:347] Connected to a master server at 127.21.12.126:45555
I20260926 16:46:10.355014 6382 heartbeater.cc:464] Registering TS with master...
I20260926 16:46:10.355634 6382 heartbeater.cc:511] Master 127.21.12.126:45555 requested a full tablet report, sending...
I20260926 16:46:10.357388 6190 ts_manager.cc:194] Registered new tserver with Master: 69e0d7aecc6148afb31c600b63b4c35d (127.21.12.65:34247)
I20260926 16:46:10.358703 6190 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.65@KRBTEST.COM'} at 127.21.12.65:42083
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab.
Entry for principal kudu/127.21.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab.
I20260926 16:46:10.368899 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.66:0
--local_ip_for_outbound_sockets=127.21.12.66
--webserver_interface=127.21.12.66
--webserver_port=0
--tserver_master_addrs=127.21.12.126:45555
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab
--principal=kudu/127.21.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:40079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:46:10.485491 6387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:10.485739 6387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:10.485869 6387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:10.489950 6387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:10.490003 6387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:10.490103 6387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.66
I20260926 16:46:10.495553 6387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:40079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab
--principal=kudu/127.21.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.66
--webserver_port=0
--tserver_master_addrs=127.21.12.126:45555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.66
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:46:10.496600 6387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:10.497627 6387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:10.507294 6393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:10.507354 6387 server_base.cc:1034] running on GCE node
W20260926 16:46:10.507296 6395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:10.507277 6392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:10.508047 6387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:10.508663 6387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:10.509815 6387 hybrid_clock.cc:648] HybridClock initialized: now 1790441170509798 us; error 32 us; skew 500 ppm
Sep 26 16:46:10 dist-test-slave-n1hw krb5kdc[6145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441170, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:46:10.512502 6387 init.cc:377] Logged in from keytab as kudu/127.21.12.66@KRBTEST.COM (short username kudu)
I20260926 16:46:10.513348 6387 webserver.cc:460] Webserver started at http://127.21.12.66:44139/ using document root <none> and password file <none>
I20260926 16:46:10.514015 6387 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:10.514101 6387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:10.514312 6387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:46:10.516355 6387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-1/data/instance:
uuid: "6bfbd664bcc24aff8ea36976b980a637"
format_stamp: "Formatted at 2026-09-26 16:46:10 on dist-test-slave-n1hw"
server_key: "73f9e899c4c452c0130837e0baf68ae0"
server_key_iv: "08fde2da326858c4f3c069cf2f2b6362"
server_key_version: "encryptionkey@0"
I20260926 16:46:10.516901 6387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance:
uuid: "6bfbd664bcc24aff8ea36976b980a637"
format_stamp: "Formatted at 2026-09-26 16:46:10 on dist-test-slave-n1hw"
server_key: "73f9e899c4c452c0130837e0baf68ae0"
server_key_iv: "08fde2da326858c4f3c069cf2f2b6362"
server_key_version: "encryptionkey@0"
I20260926 16:46:10.520473 6387 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260926 16:46:10.522796 6402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:10.524103 6387 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:46:10.524245 6387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-1/wal
uuid: "6bfbd664bcc24aff8ea36976b980a637"
format_stamp: "Formatted at 2026-09-26 16:46:10 on dist-test-slave-n1hw"
server_key: "73f9e899c4c452c0130837e0baf68ae0"
server_key_iv: "08fde2da326858c4f3c069cf2f2b6362"
server_key_version: "encryptionkey@0"
I20260926 16:46:10.524350 6387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:10.536942 6387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:10.539821 6387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:10.540021 6387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:10.540582 6387 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:46:10.541378 6387 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:46:10.541441 6387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:10.541515 6387 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:46:10.541554 6387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:10.550379 6387 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.66:45353
I20260926 16:46:10.550410 6516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.66:45353 every 8 connection(s)
I20260926 16:46:10.551525 6387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
I20260926 16:46:10.555496 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 6387
I20260926 16:46:10.555629 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance
I20260926 16:46:10.555914 21553 external_mini_cluster.cc:1428] Setting key 59d3c2b3eeee78ea39221dca90dca0ca
Sep 26 16:46:10 dist-test-slave-n1hw krb5kdc[6145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441170, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.66@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.12.67@KRBTEST.COM; defaulting to no policy
I20260926 16:46:10.566519 6229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:10.553808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:55067 (local address 127.21.12.126:45555)
0926 16:46:10.553930 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:10.553934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:10.554798 (+ 864us) server_negotiation.cc:407] Connection header received
0926 16:46:10.556488 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:10.556496 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:10.556546 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:10.556614 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:10.558285 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:10.558861 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:10.560146 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:10.560376 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:10.560492 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:10.562921 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:10.562925 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:10.562927 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:10.562960 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:10.564393 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:10.564967 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:10.564971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:10.564973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:10.565074 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:10.565382 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:10.565385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:10.565386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:10.565600 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:10.565699 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:10.566277 (+ 578us) server_negotiation.cc:299] Negotiation successful
0926 16:46:10.566375 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.67@KRBTEST.COM" created.
I20260926 16:46:10.567445 6519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:10.554198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:45555 (local address 127.21.12.66:55067)
0926 16:46:10.554645 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:10.554687 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:10.556259 (+ 1572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:10.556853 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:10.556860 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:10.557378 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:46:10.558131 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:10.558155 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:10.559000 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:10.559006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:10.560021 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:10.560030 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:10.560521 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:10.560525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:10.560701 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:10.561301 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:46:10.561317 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:46:10.562762 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:46:10.564543 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:10.564549 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:10.564564 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:10.564826 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:10.565165 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:10.565167 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:10.565169 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:10.565295 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:10.565734 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:46:10.565743 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:46:10.566053 (+ 310us) client_negotiation.cc:769] Sending connection context
0926 16:46:10.566208 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 16:46:10.566460 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":141,"threads_started":1}
I20260926 16:46:10.568600 6517 heartbeater.cc:347] Connected to a master server at 127.21.12.126:45555
I20260926 16:46:10.568845 6517 heartbeater.cc:464] Registering TS with master...
I20260926 16:46:10.569330 6517 heartbeater.cc:511] Master 127.21.12.126:45555 requested a full tablet report, sending...
I20260926 16:46:10.570439 6190 ts_manager.cc:194] Registered new tserver with Master: 6bfbd664bcc24aff8ea36976b980a637 (127.21.12.66:45353)
I20260926 16:46:10.571316 6190 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.66@KRBTEST.COM'} at 127.21.12.66:55067
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab.
Entry for principal kudu/127.21.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab.
I20260926 16:46:10.584769 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.67:0
--local_ip_for_outbound_sockets=127.21.12.67
--webserver_interface=127.21.12.67
--webserver_port=0
--tserver_master_addrs=127.21.12.126:45555
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab
--principal=kudu/127.21.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:40079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:46:10.697817 6522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:10.698073 6522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:10.698115 6522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:10.702250 6522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:10.702307 6522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:10.702370 6522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.67
I20260926 16:46:10.707559 6522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:40079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab
--principal=kudu/127.21.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.67
--webserver_port=0
--tserver_master_addrs=127.21.12.126:45555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.67
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:46:10.708537 6522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:10.709486 6522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:10.719100 6530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:10.719135 6527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:10.719254 6522 server_base.cc:1034] running on GCE node
W20260926 16:46:10.719147 6528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:10.719700 6522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:10.720417 6522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:10.721582 6522 hybrid_clock.cc:648] HybridClock initialized: now 1790441170721564 us; error 30 us; skew 500 ppm
Sep 26 16:46:10 dist-test-slave-n1hw krb5kdc[6145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441170, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:46:10.724309 6522 init.cc:377] Logged in from keytab as kudu/127.21.12.67@KRBTEST.COM (short username kudu)
I20260926 16:46:10.725193 6522 webserver.cc:460] Webserver started at http://127.21.12.67:33027/ using document root <none> and password file <none>
I20260926 16:46:10.725835 6522 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:10.725922 6522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:10.726186 6522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:46:10.728219 6522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-2/data/instance:
uuid: "c31312ed8e3449e8b2116c2ba5d2f834"
format_stamp: "Formatted at 2026-09-26 16:46:10 on dist-test-slave-n1hw"
server_key: "a9efb188c04a4b17422376cf012547f6"
server_key_iv: "1860380811704e898e930f1c34e9a0be"
server_key_version: "encryptionkey@0"
I20260926 16:46:10.728767 6522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance:
uuid: "c31312ed8e3449e8b2116c2ba5d2f834"
format_stamp: "Formatted at 2026-09-26 16:46:10 on dist-test-slave-n1hw"
server_key: "a9efb188c04a4b17422376cf012547f6"
server_key_iv: "1860380811704e898e930f1c34e9a0be"
server_key_version: "encryptionkey@0"
I20260926 16:46:10.732487 6522 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260926 16:46:10.734944 6537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:10.736068 6522 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 16:46:10.736192 6522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-2/wal
uuid: "c31312ed8e3449e8b2116c2ba5d2f834"
format_stamp: "Formatted at 2026-09-26 16:46:10 on dist-test-slave-n1hw"
server_key: "a9efb188c04a4b17422376cf012547f6"
server_key_iv: "1860380811704e898e930f1c34e9a0be"
server_key_version: "encryptionkey@0"
I20260926 16:46:10.736296 6522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:10.748721 6522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:10.751499 6522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:10.751694 6522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:10.752255 6522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:46:10.753032 6522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:46:10.753095 6522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:10.753161 6522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:46:10.753207 6522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:10.762036 6522 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.67:32909
I20260926 16:46:10.762075 6651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.67:32909 every 8 connection(s)
I20260926 16:46:10.762885 6522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
I20260926 16:46:10.772169 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 6522
I20260926 16:46:10.772300 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance
Sep 26 16:46:10 dist-test-slave-n1hw krb5kdc[6145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441170, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.67@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:46:10.772616 21553 external_mini_cluster.cc:1428] Setting key 83c59ba2ea60613d68095ce52b0f6ddc
I20260926 16:46:10.776684 6229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:10.764819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:57869 (local address 127.21.12.126:45555)
0926 16:46:10.765009 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:10.765014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:10.765881 (+ 867us) server_negotiation.cc:407] Connection header received
0926 16:46:10.767202 (+ 1321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:10.767204 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:10.767253 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:10.767330 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:10.768494 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:10.769042 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:10.770213 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:10.770426 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:10.770518 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:10.773023 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:10.773029 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:10.773036 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:10.773067 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:10.774673 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:10.775182 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:10.775185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:10.775187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:10.775265 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:10.775620 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:10.775623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:10.775624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:10.775805 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:10.775947 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:10.776475 (+ 528us) server_negotiation.cc:299] Negotiation successful
0926 16:46:10.776571 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 16:46:10.777541 6654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:10.765229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:45555 (local address 127.21.12.67:57869)
0926 16:46:10.765693 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:10.765734 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:10.767000 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:10.767527 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:10.767534 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:10.767888 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:46:10.768352 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:10.768365 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:10.769146 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:10.769148 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:10.770103 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:10.770112 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:10.770581 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:10.770584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:10.770721 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:10.771332 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:46:10.771348 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:46:10.772857 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:46:10.774785 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:10.774790 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:10.774804 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:10.775078 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:10.775352 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:10.775354 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:10.775356 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:10.775514 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:10.775941 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:46:10.775947 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:46:10.776237 (+ 290us) client_negotiation.cc:769] Sending connection context
0926 16:46:10.776397 (+ 160us) client_negotiation.cc:240] Negotiation successful
0926 16:46:10.776677 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":116,"threads_started":1}
I20260926 16:46:10.778640 6652 heartbeater.cc:347] Connected to a master server at 127.21.12.126:45555
I20260926 16:46:10.778869 6652 heartbeater.cc:464] Registering TS with master...
I20260926 16:46:10.779330 6652 heartbeater.cc:511] Master 127.21.12.126:45555 requested a full tablet report, sending...
I20260926 16:46:10.780385 6190 ts_manager.cc:194] Registered new tserver with Master: c31312ed8e3449e8b2116c2ba5d2f834 (127.21.12.67:32909)
I20260926 16:46:10.780973 6190 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.67@KRBTEST.COM'} at 127.21.12.67:57869
I20260926 16:46:10.786404 21553 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:46:10.795840 6229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:10.788276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49944 (local address 127.21.12.126:45555)
0926 16:46:10.788436 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:10.788441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:10.788628 (+ 187us) server_negotiation.cc:407] Connection header received
0926 16:46:10.788717 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:10.788719 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:10.788757 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:10.788829 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:10.789471 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:10.790052 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:10.790856 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:10.791074 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:10.791153 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:10.792508 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:10.792511 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:10.792513 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:10.792533 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:10.794171 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:10.794571 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:10.794575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:10.794576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:10.794671 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:10.794958 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:10.794961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:10.794963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:10.795184 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:10.795319 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:10.795563 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 16:46:10.795676 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 16:46:10.798683 6190 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:46:10.802050 6190 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:46:10.816022 6664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:10.810059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.67:32909 (local address 127.0.0.1:49948)
0926 16:46:10.810531 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:10.810581 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:10.810712 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:10.811510 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:10.811515 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:10.811536 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:46:10.811680 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:10.811691 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:10.812956 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:10.812959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:10.814306 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:10.814316 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:10.815449 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:10.815453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:10.815561 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:10.815607 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 16:46:10.815695 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 16:46:10.815815 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":20,"thread_start_us":80,"threads_started":1}
I20260926 16:46:10.816116 6668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:10.811230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36112 (local address 127.21.12.65:34247)
0926 16:46:10.811533 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:10.811548 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:10.811572 (+ 24us) server_negotiation.cc:407] Connection header received
0926 16:46:10.811646 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:10.811654 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:10.811838 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:10.811999 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:46:10.812212 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:10.813366 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:10.814884 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:10.815549 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:10.815669 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:10.815824 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:10.815879 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 16:46:10.815970 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":98,"threads_started":1}
I20260926 16:46:10.816071 6663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:10.810799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:34247 (local address 127.0.0.1:36112)
0926 16:46:10.811231 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:10.811245 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:10.811349 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:10.811979 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:10.811982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:10.812000 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:46:10.812093 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:10.812100 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:10.813492 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:10.813496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:10.814768 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:10.814777 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:10.815668 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:10.815672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:10.815764 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:10.815786 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 16:46:10.815850 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 16:46:10.815919 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"mutex_wait_us":2,"thread_start_us":98,"threads_started":1}
I20260926 16:46:10.816330 6666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:10.810115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:45353 (local address 127.0.0.1:41942)
0926 16:46:10.811062 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:10.811077 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:10.811157 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:10.812250 (+ 1093us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:10.812252 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:10.812267 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:46:10.812349 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:10.812354 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:10.813667 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:10.813673 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:10.814873 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:10.814883 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:10.815971 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:10.815975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:10.816066 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:10.816097 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 16:46:10.816154 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 16:46:10.816233 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":872,"thread_start_us":76,"threads_started":1}
I20260926 16:46:10.816815 6665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:10.810165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49948 (local address 127.21.12.67:32909)
0926 16:46:10.810598 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:10.811014 (+ 416us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:10.811040 (+ 26us) server_negotiation.cc:407] Connection header received
0926 16:46:10.811136 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:10.811141 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:10.811342 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:10.811527 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:46:10.811817 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:10.812819 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:10.814446 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:10.815307 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:10.816343 (+ 1036us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:10.816467 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:10.816537 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 16:46:10.816638 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":111,"threads_started":1}
I20260926 16:46:10.817093 6667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:10.810861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41942 (local address 127.21.12.66:45353)
0926 16:46:10.811828 (+ 967us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:10.811839 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:10.811861 (+ 22us) server_negotiation.cc:407] Connection header received
0926 16:46:10.811949 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:10.811954 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:10.812125 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:10.812280 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:46:10.812432 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:10.813550 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:10.814994 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:10.815842 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:10.816425 (+ 583us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:10.816588 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:10.816629 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 16:46:10.816936 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":873,"thread_start_us":127,"threads_started":1}
I20260926 16:46:10.819540 6316 tablet_service.cc:1467] Processing CreateTablet for tablet d753966ee933441fb3eaf87f46b151b7 (DEFAULT_TABLE table=test-table [id=5a27ea35ef2547c6bb7d59e0ce467ff1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:46:10.819592 6586 tablet_service.cc:1467] Processing CreateTablet for tablet d753966ee933441fb3eaf87f46b151b7 (DEFAULT_TABLE table=test-table [id=5a27ea35ef2547c6bb7d59e0ce467ff1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:46:10.819592 6451 tablet_service.cc:1467] Processing CreateTablet for tablet d753966ee933441fb3eaf87f46b151b7 (DEFAULT_TABLE table=test-table [id=5a27ea35ef2547c6bb7d59e0ce467ff1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:46:10.820549 6316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d753966ee933441fb3eaf87f46b151b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:10.820590 6586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d753966ee933441fb3eaf87f46b151b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:10.820585 6451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d753966ee933441fb3eaf87f46b151b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:10.827342 6669 tablet_bootstrap.cc:492] T d753966ee933441fb3eaf87f46b151b7 P 69e0d7aecc6148afb31c600b63b4c35d: Bootstrap starting.
I20260926 16:46:10.827939 6670 tablet_bootstrap.cc:492] T d753966ee933441fb3eaf87f46b151b7 P 6bfbd664bcc24aff8ea36976b980a637: Bootstrap starting.
I20260926 16:46:10.828210 6671 tablet_bootstrap.cc:492] T d753966ee933441fb3eaf87f46b151b7 P c31312ed8e3449e8b2116c2ba5d2f834: Bootstrap starting.
I20260926 16:46:10.830127 6671 tablet_bootstrap.cc:654] T d753966ee933441fb3eaf87f46b151b7 P c31312ed8e3449e8b2116c2ba5d2f834: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:10.830180 6669 tablet_bootstrap.cc:654] T d753966ee933441fb3eaf87f46b151b7 P 69e0d7aecc6148afb31c600b63b4c35d: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:10.830703 6670 tablet_bootstrap.cc:654] T d753966ee933441fb3eaf87f46b151b7 P 6bfbd664bcc24aff8ea36976b980a637: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:10.831275 6671 log.cc:824] T d753966ee933441fb3eaf87f46b151b7 P c31312ed8e3449e8b2116c2ba5d2f834: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:10.831487 6669 log.cc:824] T d753966ee933441fb3eaf87f46b151b7 P 69e0d7aecc6148afb31c600b63b4c35d: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:10.831487 6670 log.cc:824] T d753966ee933441fb3eaf87f46b151b7 P 6bfbd664bcc24aff8ea36976b980a637: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:10.833228 6671 tablet_bootstrap.cc:492] T d753966ee933441fb3eaf87f46b151b7 P c31312ed8e3449e8b2116c2ba5d2f834: No bootstrap required, opened a new log
I20260926 16:46:10.833428 6671 ts_tablet_manager.cc:1397] T d753966ee933441fb3eaf87f46b151b7 P c31312ed8e3449e8b2116c2ba5d2f834: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 16:46:10.833712 6670 tablet_bootstrap.cc:492] T d753966ee933441fb3eaf87f46b151b7 P 6bfbd664bcc24aff8ea36976b980a637: No bootstrap required, opened a new log
I20260926 16:46:10.833724 6669 tablet_bootstrap.cc:492] T d753966ee933441fb3eaf87f46b151b7 P 69e0d7aecc6148afb31c600b63b4c35d: No bootstrap required, opened a new log
I20260926 16:46:10.833920 6670 ts_tablet_manager.cc:1397] T d753966ee933441fb3eaf87f46b151b7 P 6bfbd664bcc24aff8ea36976b980a637: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260926 16:46:10.833920 6669 ts_tablet_manager.cc:1397] T d753966ee933441fb3eaf87f46b151b7 P 69e0d7aecc6148afb31c600b63b4c35d: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260926 16:46:10.836331 6671 raft_consensus.cc:348] T d753966ee933441fb3eaf87f46b151b7 P c31312ed8e3449e8b2116c2ba5d2f834 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31312ed8e3449e8b2116c2ba5d2f834" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 32909 } } peers { permanent_uuid: "6bfbd664bcc24aff8ea36976b980a637" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 45353 } } peers { permanent_uuid: "69e0d7aecc6148afb31c600b63b4c35d" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 34247 } }
I20260926 16:46:10.836583 6671 raft_consensus.cc:374] T d753966ee933441fb3eaf87f46b151b7 P c31312ed8e3449e8b2116c2ba5d2f834 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:46:10.836637 6671 raft_consensus.cc:729] T d753966ee933441fb3eaf87f46b151b7 P c31312ed8e3449e8b2116c2ba5d2f834 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c31312ed8e3449e8b2116c2ba5d2f834, State: Initialized, Role: FOLLOWER
I20260926 16:46:10.837105 6671 consensus_queue.cc:260] T d753966ee933441fb3eaf87f46b151b7 P c31312ed8e3449e8b2116c2ba5d2f834 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31312ed8e3449e8b2116c2ba5d2f834" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 32909 } } peers { permanent_uuid: "6bfbd664bcc24aff8ea36976b980a637" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 45353 } } peers { permanent_uuid: "69e0d7aecc6148afb31c600b63b4c35d" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 34247 } }
I20260926 16:46:10.837787 6670 raft_consensus.cc:348] T d753966ee933441fb3eaf87f46b151b7 P 6bfbd664bcc24aff8ea36976b980a637 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31312ed8e3449e8b2116c2ba5d2f834" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 32909 } } peers { permanent_uuid: "6bfbd664bcc24aff8ea36976b980a637" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 45353 } } peers { permanent_uuid: "69e0d7aecc6148afb31c600b63b4c35d" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 34247 } }
I20260926 16:46:10.837787 6669 raft_consensus.cc:348] T d753966ee933441fb3eaf87f46b151b7 P 69e0d7aecc6148afb31c600b63b4c35d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31312ed8e3449e8b2116c2ba5d2f834" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 32909 } } peers { permanent_uuid: "6bfbd664bcc24aff8ea36976b980a637" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 45353 } } peers { permanent_uuid: "69e0d7aecc6148afb31c600b63b4c35d" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 34247 } }
I20260926 16:46:10.838066 6669 raft_consensus.cc:374] T d753966ee933441fb3eaf87f46b151b7 P 69e0d7aecc6148afb31c600b63b4c35d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:46:10.838066 6670 raft_consensus.cc:374] T d753966ee933441fb3eaf87f46b151b7 P 6bfbd664bcc24aff8ea36976b980a637 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:46:10.838119 6671 ts_tablet_manager.cc:1428] T d753966ee933441fb3eaf87f46b151b7 P c31312ed8e3449e8b2116c2ba5d2f834: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 16:46:10.838129 6669 raft_consensus.cc:729] T d753966ee933441fb3eaf87f46b151b7 P 69e0d7aecc6148afb31c600b63b4c35d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69e0d7aecc6148afb31c600b63b4c35d, State: Initialized, Role: FOLLOWER
I20260926 16:46:10.838129 6670 raft_consensus.cc:729] T d753966ee933441fb3eaf87f46b151b7 P 6bfbd664bcc24aff8ea36976b980a637 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bfbd664bcc24aff8ea36976b980a637, State: Initialized, Role: FOLLOWER
I20260926 16:46:10.838409 6652 heartbeater.cc:503] Master 127.21.12.126:45555 was elected leader, sending a full tablet report...
I20260926 16:46:10.838580 6670 consensus_queue.cc:260] T d753966ee933441fb3eaf87f46b151b7 P 6bfbd664bcc24aff8ea36976b980a637 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31312ed8e3449e8b2116c2ba5d2f834" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 32909 } } peers { permanent_uuid: "6bfbd664bcc24aff8ea36976b980a637" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 45353 } } peers { permanent_uuid: "69e0d7aecc6148afb31c600b63b4c35d" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 34247 } }
I20260926 16:46:10.838580 6669 consensus_queue.cc:260] T d753966ee933441fb3eaf87f46b151b7 P 69e0d7aecc6148afb31c600b63b4c35d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31312ed8e3449e8b2116c2ba5d2f834" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 32909 } } peers { permanent_uuid: "6bfbd664bcc24aff8ea36976b980a637" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 45353 } } peers { permanent_uuid: "69e0d7aecc6148afb31c600b63b4c35d" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 34247 } }
I20260926 16:46:10.839749 6669 ts_tablet_manager.cc:1428] T d753966ee933441fb3eaf87f46b151b7 P 69e0d7aecc6148afb31c600b63b4c35d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 16:46:10.839749 6670 ts_tablet_manager.cc:1428] T d753966ee933441fb3eaf87f46b151b7 P 6bfbd664bcc24aff8ea36976b980a637: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260926 16:46:10.840072 6382 heartbeater.cc:503] Master 127.21.12.126:45555 was elected leader, sending a full tablet report...
I20260926 16:46:10.840099 6517 heartbeater.cc:503] Master 127.21.12.126:45555 was elected leader, sending a full tablet report...
I20260926 16:46:10.850760 6675 raft_consensus.cc:482] T d753966ee933441fb3eaf87f46b151b7 P c31312ed8e3449e8b2116c2ba5d2f834 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:46:10.850929 6675 raft_consensus.cc:504] T d753966ee933441fb3eaf87f46b151b7 P c31312ed8e3449e8b2116c2ba5d2f834 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31312ed8e3449e8b2116c2ba5d2f834" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 32909 } } peers { permanent_uuid: "6bfbd664bcc24aff8ea36976b980a637" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 45353 } } peers { permanent_uuid: "69e0d7aecc6148afb31c600b63b4c35d" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 34247 } }
I20260926 16:46:10.851987 6675 leader_election.cc:288] T d753966ee933441fb3eaf87f46b151b7 P c31312ed8e3449e8b2116c2ba5d2f834 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bfbd664bcc24aff8ea36976b980a637 (127.21.12.66:45353), 69e0d7aecc6148afb31c600b63b4c35d (127.21.12.65:34247)
I20260926 16:46:10.856781 6654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:10.852153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:45353 (local address 127.21.12.67:60665)
0926 16:46:10.852351 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:10.852368 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:10.852452 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:10.852797 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:10.852799 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:10.852812 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:46:10.852973 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:10.852982 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:10.854019 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:10.854023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:10.855490 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:10.855503 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:10.856367 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:10.856370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:10.856537 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:10.856557 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 16:46:10.856621 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 16:46:10.856674 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":207}
I20260926 16:46:10.856801 6667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:10.852242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:60665 (local address 127.21.12.66:45353)
0926 16:46:10.852375 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:10.852381 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:10.852404 (+ 23us) server_negotiation.cc:407] Connection header received
0926 16:46:10.852613 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:10.852616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:10.852689 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:10.852764 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:46:10.853107 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:10.853901 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:10.855631 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:10.856250 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:10.856372 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:10.856424 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:10.856648 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 16:46:10.856709 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 16:46:10.857369 6679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:10.852239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:34247 (local address 127.21.12.67:48873)
0926 16:46:10.852585 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:10.852597 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:10.852680 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:10.852885 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:10.852887 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:10.852900 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:46:10.853012 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:10.853021 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:10.854030 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:10.854033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:10.855520 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:10.855529 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:10.856356 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:10.856360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:10.857064 (+ 704us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:10.857081 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 16:46:10.857161 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 16:46:10.857226 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":114,"thread_start_us":117,"threads_started":1}
I20260926 16:46:10.857352 6668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:10.852335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:48873 (local address 127.21.12.65:34247)
0926 16:46:10.852453 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:10.852458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:10.852671 (+ 213us) server_negotiation.cc:407] Connection header received
0926 16:46:10.852753 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:10.852758 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:10.852801 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:10.852872 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:46:10.853109 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:10.853948 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:10.855631 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:10.856227 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:10.856372 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:10.856424 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:10.857177 (+ 753us) server_negotiation.cc:299] Negotiation successful
0926 16:46:10.857249 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 16:46:10.857366 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d753966ee933441fb3eaf87f46b151b7" candidate_uuid: "c31312ed8e3449e8b2116c2ba5d2f834" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bfbd664bcc24aff8ea36976b980a637" is_pre_election: true
I20260926 16:46:10.857697 6471 raft_consensus.cc:2454] T d753966ee933441fb3eaf87f46b151b7 P 6bfbd664bcc24aff8ea36976b980a637 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c31312ed8e3449e8b2116c2ba5d2f834 in term 0.
I20260926 16:46:10.858099 6336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d753966ee933441fb3eaf87f46b151b7" candidate_uuid: "c31312ed8e3449e8b2116c2ba5d2f834" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69e0d7aecc6148afb31c600b63b4c35d" is_pre_election: true
I20260926 16:46:10.858294 6540 leader_election.cc:302] T d753966ee933441fb3eaf87f46b151b7 P c31312ed8e3449e8b2116c2ba5d2f834 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bfbd664bcc24aff8ea36976b980a637, c31312ed8e3449e8b2116c2ba5d2f834; no voters:
I20260926 16:46:10.858479 6336 raft_consensus.cc:2454] T d753966ee933441fb3eaf87f46b151b7 P 69e0d7aecc6148afb31c600b63b4c35d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c31312ed8e3449e8b2116c2ba5d2f834 in term 0.
I20260926 16:46:10.858572 6675 raft_consensus.cc:2781] T d753966ee933441fb3eaf87f46b151b7 P c31312ed8e3449e8b2116c2ba5d2f834 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:46:10.858654 6675 raft_consensus.cc:482] T d753966ee933441fb3eaf87f46b151b7 P c31312ed8e3449e8b2116c2ba5d2f834 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:46:10.858719 6675 raft_consensus.cc:3037] T d753966ee933441fb3eaf87f46b151b7 P c31312ed8e3449e8b2116c2ba5d2f834 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:10.859779 6675 raft_consensus.cc:504] T d753966ee933441fb3eaf87f46b151b7 P c31312ed8e3449e8b2116c2ba5d2f834 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31312ed8e3449e8b2116c2ba5d2f834" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 32909 } } peers { permanent_uuid: "6bfbd664bcc24aff8ea36976b980a637" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 45353 } } peers { permanent_uuid: "69e0d7aecc6148afb31c600b63b4c35d" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 34247 } }
I20260926 16:46:10.860142 6675 leader_election.cc:288] T d753966ee933441fb3eaf87f46b151b7 P c31312ed8e3449e8b2116c2ba5d2f834 [CANDIDATE]: Term 1 election: Requested vote from peers 6bfbd664bcc24aff8ea36976b980a637 (127.21.12.66:45353), 69e0d7aecc6148afb31c600b63b4c35d (127.21.12.65:34247)
I20260926 16:46:10.860446 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d753966ee933441fb3eaf87f46b151b7" candidate_uuid: "c31312ed8e3449e8b2116c2ba5d2f834" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bfbd664bcc24aff8ea36976b980a637"
I20260926 16:46:10.860536 6336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d753966ee933441fb3eaf87f46b151b7" candidate_uuid: "c31312ed8e3449e8b2116c2ba5d2f834" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69e0d7aecc6148afb31c600b63b4c35d"
I20260926 16:46:10.860554 6471 raft_consensus.cc:3037] T d753966ee933441fb3eaf87f46b151b7 P 6bfbd664bcc24aff8ea36976b980a637 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:10.860651 6336 raft_consensus.cc:3037] T d753966ee933441fb3eaf87f46b151b7 P 69e0d7aecc6148afb31c600b63b4c35d [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:10.861562 6471 raft_consensus.cc:2454] T d753966ee933441fb3eaf87f46b151b7 P 6bfbd664bcc24aff8ea36976b980a637 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c31312ed8e3449e8b2116c2ba5d2f834 in term 1.
I20260926 16:46:10.861654 6336 raft_consensus.cc:2454] T d753966ee933441fb3eaf87f46b151b7 P 69e0d7aecc6148afb31c600b63b4c35d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c31312ed8e3449e8b2116c2ba5d2f834 in term 1.
I20260926 16:46:10.861858 6540 leader_election.cc:302] T d753966ee933441fb3eaf87f46b151b7 P c31312ed8e3449e8b2116c2ba5d2f834 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bfbd664bcc24aff8ea36976b980a637, c31312ed8e3449e8b2116c2ba5d2f834; no voters:
I20260926 16:46:10.862015 6675 raft_consensus.cc:2781] T d753966ee933441fb3eaf87f46b151b7 P c31312ed8e3449e8b2116c2ba5d2f834 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:46:10.862188 6675 raft_consensus.cc:686] T d753966ee933441fb3eaf87f46b151b7 P c31312ed8e3449e8b2116c2ba5d2f834 [term 1 LEADER]: Becoming Leader. State: Replica: c31312ed8e3449e8b2116c2ba5d2f834, State: Running, Role: LEADER
I20260926 16:46:10.862488 6675 consensus_queue.cc:237] T d753966ee933441fb3eaf87f46b151b7 P c31312ed8e3449e8b2116c2ba5d2f834 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31312ed8e3449e8b2116c2ba5d2f834" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 32909 } } peers { permanent_uuid: "6bfbd664bcc24aff8ea36976b980a637" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 45353 } } peers { permanent_uuid: "69e0d7aecc6148afb31c600b63b4c35d" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 34247 } }
I20260926 16:46:10.865360 6189 catalog_manager.cc:5495] T d753966ee933441fb3eaf87f46b151b7 P c31312ed8e3449e8b2116c2ba5d2f834 reported cstate change: term changed from 0 to 1, leader changed from <none> to c31312ed8e3449e8b2116c2ba5d2f834 (127.21.12.67). New cstate: current_term: 1 leader_uuid: "c31312ed8e3449e8b2116c2ba5d2f834" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31312ed8e3449e8b2116c2ba5d2f834" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 32909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bfbd664bcc24aff8ea36976b980a637" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 45353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69e0d7aecc6148afb31c600b63b4c35d" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 34247 } health_report { overall_health: UNKNOWN } } }
I20260926 16:46:10.878397 6665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:10.874828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49956 (local address 127.21.12.67:32909)
0926 16:46:10.874951 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:10.874957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:10.874972 (+ 15us) server_negotiation.cc:407] Connection header received
0926 16:46:10.875160 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:10.875162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:10.875210 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:10.875311 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:46:10.875572 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:10.876095 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:10.876983 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:10.877189 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:10.877267 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:10.877550 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:46:10.878009 (+ 459us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:46:10.878099 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:10.878235 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 16:46:10.878279 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 16:46:10.884063 6471 raft_consensus.cc:1261] T d753966ee933441fb3eaf87f46b151b7 P 6bfbd664bcc24aff8ea36976b980a637 [term 1 FOLLOWER]: Refusing update from remote peer c31312ed8e3449e8b2116c2ba5d2f834: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:46:10.884063 6336 raft_consensus.cc:1261] T d753966ee933441fb3eaf87f46b151b7 P 69e0d7aecc6148afb31c600b63b4c35d [term 1 FOLLOWER]: Refusing update from remote peer c31312ed8e3449e8b2116c2ba5d2f834: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:46:10.884768 6680 consensus_queue.cc:1035] T d753966ee933441fb3eaf87f46b151b7 P c31312ed8e3449e8b2116c2ba5d2f834 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bfbd664bcc24aff8ea36976b980a637" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 45353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:46:10.884994 6675 consensus_queue.cc:1035] T d753966ee933441fb3eaf87f46b151b7 P c31312ed8e3449e8b2116c2ba5d2f834 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69e0d7aecc6148afb31c600b63b4c35d" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:46:10.895747 6684 mvcc.cc:205] Tried to move back new op lower bound from 7333647035934228480 to 7333647035855286272. Current Snapshot: MvccSnapshot[applied={T|T < 7333647035934228480}]
I20260926 16:46:10.900892 6188 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:46:10.901041 6188 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:46:10.903427 6188 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e49f63a3db743b7a234d1cfd0b9359e: Sending DeleteTablet for 3 replicas of tablet d753966ee933441fb3eaf87f46b151b7
I20260926 16:46:10.904299 6586 tablet_service.cc:1514] Processing DeleteTablet for tablet d753966ee933441fb3eaf87f46b151b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:46:10 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:49948
I20260926 16:46:10.904394 6451 tablet_service.cc:1514] Processing DeleteTablet for tablet d753966ee933441fb3eaf87f46b151b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:46:10 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:41942
I20260926 16:46:10.904578 6316 tablet_service.cc:1514] Processing DeleteTablet for tablet d753966ee933441fb3eaf87f46b151b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:46:10 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:36112
I20260926 16:46:10.904816 6692 tablet_replica.cc:330] T d753966ee933441fb3eaf87f46b151b7 P c31312ed8e3449e8b2116c2ba5d2f834: stopping tablet replica
I20260926 16:46:10.904846 6693 tablet_replica.cc:330] T d753966ee933441fb3eaf87f46b151b7 P 6bfbd664bcc24aff8ea36976b980a637: stopping tablet replica
I20260926 16:46:10.905138 6692 raft_consensus.cc:2229] T d753966ee933441fb3eaf87f46b151b7 P c31312ed8e3449e8b2116c2ba5d2f834 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:46:10.905143 6693 raft_consensus.cc:2229] T d753966ee933441fb3eaf87f46b151b7 P 6bfbd664bcc24aff8ea36976b980a637 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:46:10.905517 6692 raft_consensus.cc:2258] T d753966ee933441fb3eaf87f46b151b7 P c31312ed8e3449e8b2116c2ba5d2f834 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:46:10.905928 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 6252
I20260926 16:46:10.905923 6693 raft_consensus.cc:2258] T d753966ee933441fb3eaf87f46b151b7 P 6bfbd664bcc24aff8ea36976b980a637 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:46:10.908321 6693 ts_tablet_manager.cc:1905] T d753966ee933441fb3eaf87f46b151b7 P 6bfbd664bcc24aff8ea36976b980a637: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:46:10.909389 6692 ts_tablet_manager.cc:1905] T d753966ee933441fb3eaf87f46b151b7 P c31312ed8e3449e8b2116c2ba5d2f834: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:46:10.911118 6693 ts_tablet_manager.cc:1918] T d753966ee933441fb3eaf87f46b151b7 P 6bfbd664bcc24aff8ea36976b980a637: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:46:10.911217 6693 log.cc:1196] T d753966ee933441fb3eaf87f46b151b7 P 6bfbd664bcc24aff8ea36976b980a637: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-1/wal/wals/d753966ee933441fb3eaf87f46b151b7
I20260926 16:46:10.911546 6693 ts_tablet_manager.cc:1939] T d753966ee933441fb3eaf87f46b151b7 P 6bfbd664bcc24aff8ea36976b980a637: Deleting consensus metadata
I20260926 16:46:10.911983 6692 ts_tablet_manager.cc:1918] T d753966ee933441fb3eaf87f46b151b7 P c31312ed8e3449e8b2116c2ba5d2f834: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:46:10.912092 6692 log.cc:1196] T d753966ee933441fb3eaf87f46b151b7 P c31312ed8e3449e8b2116c2ba5d2f834: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440924937351-21553-0/minicluster-data/ts-2/wal/wals/d753966ee933441fb3eaf87f46b151b7
I20260926 16:46:10.912456 6692 ts_tablet_manager.cc:1939] T d753966ee933441fb3eaf87f46b151b7 P c31312ed8e3449e8b2116c2ba5d2f834: Deleting consensus metadata
I20260926 16:46:10.912832 6176 catalog_manager.cc:4841] TS 6bfbd664bcc24aff8ea36976b980a637 (127.21.12.66:45353): tablet d753966ee933441fb3eaf87f46b151b7 (table test-table [id=5a27ea35ef2547c6bb7d59e0ce467ff1]) successfully deleted
I20260926 16:46:10.913667 6177 catalog_manager.cc:4841] TS c31312ed8e3449e8b2116c2ba5d2f834 (127.21.12.67:32909): tablet d753966ee933441fb3eaf87f46b151b7 (table test-table [id=5a27ea35ef2547c6bb7d59e0ce467ff1]) successfully deleted
W20260926 16:46:10.916882 6177 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.12.65:34247 (error 108)
I20260926 16:46:10.916952 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 6387
I20260926 16:46:10.918666 6663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:10.918246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:34247 (local address 127.0.0.1:36116)
0926 16:46:10.918470 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:10.918556 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.65:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260926 16:46:10.918828 6177 catalog_manager.cc:4584] TS 69e0d7aecc6148afb31c600b63b4c35d (127.21.12.65:34247): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d753966ee933441fb3eaf87f46b151b7: Network error: Client connection negotiation failed: client connection to 127.21.12.65:34247: connect: Connection refused (error 111)
I20260926 16:46:10.924964 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 6522
I20260926 16:46:10.930891 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 6158
2026-09-26T16:46:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2232 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T16:46:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:46:10Z Disabled control of system clock
I20260926 16:46:10.961648 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:33005
--webserver_interface=127.21.12.126
--webserver_port=0
--builtin_ntp_servers=127.21.12.84:46001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:33005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 16:46:11.071139 6699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:11.071416 6699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:11.071452 6699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:11.075402 6699 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:46:11.075452 6699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:11.075469 6699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:11.075489 6699 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:46:11.075505 6699 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:46:11.080302 6699 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:46001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:33005
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:33005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:46:11.081342 6699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:11.082715 6699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:11.091131 6705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:11.091142 6704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:11.091261 6699 server_base.cc:1034] running on GCE node
W20260926 16:46:11.091156 6707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:11.092003 6699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:11.093123 6699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:11.094312 6699 hybrid_clock.cc:648] HybridClock initialized: now 1790441171094291 us; error 34 us; skew 500 ppm
I20260926 16:46:11.096272 6699 webserver.cc:460] Webserver started at http://127.21.12.126:36217/ using document root <none> and password file <none>
I20260926 16:46:11.096858 6699 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:11.096959 6699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:11.097184 6699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:46:11.099269 6699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "0469d4f67666489e9581eeb0d0cff14d"
format_stamp: "Formatted at 2026-09-26 16:46:11 on dist-test-slave-n1hw"
server_key: "64b6ae538027da14dc10545979c687d6"
server_key_iv: "71ac8f84795f87f3267ea3a9605c4350"
server_key_version: "encryptionkey@0"
I20260926 16:46:11.099833 6699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "0469d4f67666489e9581eeb0d0cff14d"
format_stamp: "Formatted at 2026-09-26 16:46:11 on dist-test-slave-n1hw"
server_key: "64b6ae538027da14dc10545979c687d6"
server_key_iv: "71ac8f84795f87f3267ea3a9605c4350"
server_key_version: "encryptionkey@0"
I20260926 16:46:11.103492 6699 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 16:46:11.105979 6713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:11.107285 6699 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:46:11.107439 6699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "0469d4f67666489e9581eeb0d0cff14d"
format_stamp: "Formatted at 2026-09-26 16:46:11 on dist-test-slave-n1hw"
server_key: "64b6ae538027da14dc10545979c687d6"
server_key_iv: "71ac8f84795f87f3267ea3a9605c4350"
server_key_version: "encryptionkey@0"
I20260926 16:46:11.107555 6699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:11.120014 6699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:11.120653 6699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:11.120839 6699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:11.127987 6699 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.126:33005
I20260926 16:46:11.128000 6765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.126:33005 every 8 connection(s)
I20260926 16:46:11.128899 6699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:46:11.131923 6766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:11.137214 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 6699
I20260926 16:46:11.137354 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:46:11.137372 6766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0469d4f67666489e9581eeb0d0cff14d: Bootstrap starting.
I20260926 16:46:11.137655 21553 external_mini_cluster.cc:1428] Setting key 4e9c8479aa0df03ef63a7e7353ecadfc
I20260926 16:46:11.140033 6766 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0469d4f67666489e9581eeb0d0cff14d: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:11.140801 6766 log.cc:824] T 00000000000000000000000000000000 P 0469d4f67666489e9581eeb0d0cff14d: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:11.142765 6766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0469d4f67666489e9581eeb0d0cff14d: No bootstrap required, opened a new log
I20260926 16:46:11.146936 6766 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0469d4f67666489e9581eeb0d0cff14d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0469d4f67666489e9581eeb0d0cff14d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 33005 } }
I20260926 16:46:11.147158 6766 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0469d4f67666489e9581eeb0d0cff14d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:46:11.147221 6766 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0469d4f67666489e9581eeb0d0cff14d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0469d4f67666489e9581eeb0d0cff14d, State: Initialized, Role: FOLLOWER
I20260926 16:46:11.147401 6769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:11.139063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49242 (local address 127.21.12.126:33005)
0926 16:46:11.139758 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:11.139772 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:11.139809 (+ 37us) server_negotiation.cc:407] Connection header received
0926 16:46:11.140485 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:11.140504 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:11.140959 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:11.141310 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:11.141798 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:11.142762 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:11.143977 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:11.144470 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:11.144604 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:11.145507 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:11.145511 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:11.145523 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:46:11.145970 (+ 447us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:11.146026 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:46:11.146046 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:11.146124 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:11.146296 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 16:46:11.146609 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":130,"threads_started":1}
I20260926 16:46:11.147702 6766 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0469d4f67666489e9581eeb0d0cff14d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0469d4f67666489e9581eeb0d0cff14d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 33005 } }
I20260926 16:46:11.147841 6766 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0469d4f67666489e9581eeb0d0cff14d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:46:11.147920 6766 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0469d4f67666489e9581eeb0d0cff14d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:46:11.148042 6766 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0469d4f67666489e9581eeb0d0cff14d [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:11.148893 6766 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0469d4f67666489e9581eeb0d0cff14d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0469d4f67666489e9581eeb0d0cff14d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 33005 } }
I20260926 16:46:11.149230 6766 leader_election.cc:302] T 00000000000000000000000000000000 P 0469d4f67666489e9581eeb0d0cff14d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0469d4f67666489e9581eeb0d0cff14d; no voters:
I20260926 16:46:11.149523 6766 leader_election.cc:288] T 00000000000000000000000000000000 P 0469d4f67666489e9581eeb0d0cff14d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:46:11.149716 6771 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0469d4f67666489e9581eeb0d0cff14d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:46:11.149977 6771 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0469d4f67666489e9581eeb0d0cff14d [term 1 LEADER]: Becoming Leader. State: Replica: 0469d4f67666489e9581eeb0d0cff14d, State: Running, Role: LEADER
I20260926 16:46:11.150236 6771 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0469d4f67666489e9581eeb0d0cff14d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0469d4f67666489e9581eeb0d0cff14d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 33005 } }
I20260926 16:46:11.150761 6766 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0469d4f67666489e9581eeb0d0cff14d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:46:11.151834 6772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0469d4f67666489e9581eeb0d0cff14d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0469d4f67666489e9581eeb0d0cff14d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0469d4f67666489e9581eeb0d0cff14d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 33005 } } }
I20260926 16:46:11.151878 6773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0469d4f67666489e9581eeb0d0cff14d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0469d4f67666489e9581eeb0d0cff14d. Latest consensus state: current_term: 1 leader_uuid: "0469d4f67666489e9581eeb0d0cff14d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0469d4f67666489e9581eeb0d0cff14d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 33005 } } }
I20260926 16:46:11.151971 6773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0469d4f67666489e9581eeb0d0cff14d [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:11.151971 6772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0469d4f67666489e9581eeb0d0cff14d [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:11.152410 6780 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:46:11.155450 6780 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:46:11.160892 6780 catalog_manager.cc:1347] Generated new cluster ID: 771590a7998546abadda97f305603c6f
I20260926 16:46:11.160966 6780 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:46:11.187989 6780 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:46:11.189270 6780 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:46:11.203910 6780 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0469d4f67666489e9581eeb0d0cff14d: Generated new TSK 0
I20260926 16:46:11.204690 6780 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:46:11.216020 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.65:0
--local_ip_for_outbound_sockets=127.21.12.65
--webserver_interface=127.21.12.65
--webserver_port=0
--tserver_master_addrs=127.21.12.126:33005
--builtin_ntp_servers=127.21.12.84:46001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 16:46:11.330340 6790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:11.330593 6790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:11.330649 6790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:11.334725 6790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:11.334780 6790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:11.334864 6790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.65
I20260926 16:46:11.339663 6790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:46001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.65
--webserver_port=0
--tserver_master_addrs=127.21.12.126:33005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.65
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:46:11.340725 6790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:11.342207 6790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:11.351890 6796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:11.351894 6795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:11.352090 6798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:11.352304 6790 server_base.cc:1034] running on GCE node
I20260926 16:46:11.352824 6790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:11.353513 6790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:11.354691 6790 hybrid_clock.cc:648] HybridClock initialized: now 1790441171354664 us; error 42 us; skew 500 ppm
I20260926 16:46:11.356630 6790 webserver.cc:460] Webserver started at http://127.21.12.65:38277/ using document root <none> and password file <none>
I20260926 16:46:11.357209 6790 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:11.357298 6790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:11.357518 6790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:46:11.359576 6790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-0/data/instance:
uuid: "5499a3f938df46899e5be26effa512c3"
format_stamp: "Formatted at 2026-09-26 16:46:11 on dist-test-slave-n1hw"
server_key: "49f1be24267a69b8c26099f1690df36e"
server_key_iv: "7b1d2d8a6eafde827bf2395c7402e32d"
server_key_version: "encryptionkey@0"
I20260926 16:46:11.360145 6790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance:
uuid: "5499a3f938df46899e5be26effa512c3"
format_stamp: "Formatted at 2026-09-26 16:46:11 on dist-test-slave-n1hw"
server_key: "49f1be24267a69b8c26099f1690df36e"
server_key_iv: "7b1d2d8a6eafde827bf2395c7402e32d"
server_key_version: "encryptionkey@0"
I20260926 16:46:11.363870 6790 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 16:46:11.366307 6804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:11.367529 6790 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:46:11.367661 6790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-0/wal
uuid: "5499a3f938df46899e5be26effa512c3"
format_stamp: "Formatted at 2026-09-26 16:46:11 on dist-test-slave-n1hw"
server_key: "49f1be24267a69b8c26099f1690df36e"
server_key_iv: "7b1d2d8a6eafde827bf2395c7402e32d"
server_key_version: "encryptionkey@0"
I20260926 16:46:11.367774 6790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:11.384886 6790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:11.385643 6790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:11.385892 6790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:11.386482 6790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:46:11.387390 6790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:46:11.387466 6790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:11.387537 6790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:46:11.387578 6790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:11.396145 6790 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.65:46705
I20260926 16:46:11.396180 6918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.65:46705 every 8 connection(s)
I20260926 16:46:11.397034 6790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
I20260926 16:46:11.402911 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 6790
I20260926 16:46:11.403025 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance
I20260926 16:46:11.403323 21553 external_mini_cluster.cc:1428] Setting key 63db940e0c504392e84ab3db4327d944
I20260926 16:46:11.405583 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.66:0
--local_ip_for_outbound_sockets=127.21.12.66
--webserver_interface=127.21.12.66
--webserver_port=0
--tserver_master_addrs=127.21.12.126:33005
--builtin_ntp_servers=127.21.12.84:46001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 16:46:11.406980 6769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:11.398860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:48017 (local address 127.21.12.126:33005)
0926 16:46:11.398990 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:11.398993 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:11.400225 (+ 1232us) server_negotiation.cc:407] Connection header received
0926 16:46:11.401684 (+ 1459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:11.401686 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:11.401733 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:11.401825 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:11.402771 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:11.403404 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:11.404852 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:11.405148 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:11.405268 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:11.406450 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:11.406458 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:11.406464 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:46:11.406484 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:11.406512 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:46:11.406527 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:11.406613 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:11.406782 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 16:46:11.406845 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 16:46:11.407869 6921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:11.399239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:33005 (local address 127.21.12.65:48017)
0926 16:46:11.400063 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:11.400107 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:11.401463 (+ 1356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:11.401983 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:11.401990 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:11.402076 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:46:11.402613 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:11.402636 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:11.403523 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:11.403527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:11.404714 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:11.404724 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:11.405305 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:11.405308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:11.405455 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:11.406187 (+ 732us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:46:11.406188 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:46:11.406228 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:46:11.406292 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:46:11.406632 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:46:11.406640 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:46:11.406645 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 16:46:11.406778 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 16:46:11.407069 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":110,"threads_started":1}
I20260926 16:46:11.409077 6919 heartbeater.cc:347] Connected to a master server at 127.21.12.126:33005
I20260926 16:46:11.409317 6919 heartbeater.cc:464] Registering TS with master...
I20260926 16:46:11.409904 6919 heartbeater.cc:511] Master 127.21.12.126:33005 requested a full tablet report, sending...
I20260926 16:46:11.411471 6730 ts_manager.cc:194] Registered new tserver with Master: 5499a3f938df46899e5be26effa512c3 (127.21.12.65:46705)
I20260926 16:46:11.412665 6730 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.12.65:48017
W20260926 16:46:11.520026 6922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:11.520330 6922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:11.520406 6922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:11.524533 6922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:11.524610 6922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:11.524699 6922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.66
I20260926 16:46:11.529392 6922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:46001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.66
--webserver_port=0
--tserver_master_addrs=127.21.12.126:33005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.66
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:46:11.530522 6922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:11.531991 6922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 16:46:11.541618 6922 server_base.cc:1034] running on GCE node
W20260926 16:46:11.541601 6927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:11.541548 6928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:11.541648 6930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:11.542253 6922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:11.542901 6922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:11.544080 6922 hybrid_clock.cc:648] HybridClock initialized: now 1790441171544062 us; error 27 us; skew 500 ppm
I20260926 16:46:11.545958 6922 webserver.cc:460] Webserver started at http://127.21.12.66:35149/ using document root <none> and password file <none>
I20260926 16:46:11.546509 6922 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:11.546571 6922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:11.546860 6922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:46:11.549031 6922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-1/data/instance:
uuid: "98a67e69247a4df897a1fdf01218a56e"
format_stamp: "Formatted at 2026-09-26 16:46:11 on dist-test-slave-n1hw"
server_key: "25b8a6dd5ba9c672abb2cbb5200a6efd"
server_key_iv: "7f2dc4078e70b573be64fd094cee702b"
server_key_version: "encryptionkey@0"
I20260926 16:46:11.549628 6922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance:
uuid: "98a67e69247a4df897a1fdf01218a56e"
format_stamp: "Formatted at 2026-09-26 16:46:11 on dist-test-slave-n1hw"
server_key: "25b8a6dd5ba9c672abb2cbb5200a6efd"
server_key_iv: "7f2dc4078e70b573be64fd094cee702b"
server_key_version: "encryptionkey@0"
I20260926 16:46:11.553635 6922 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 16:46:11.556016 6936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:11.557212 6922 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:46:11.557338 6922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-1/wal
uuid: "98a67e69247a4df897a1fdf01218a56e"
format_stamp: "Formatted at 2026-09-26 16:46:11 on dist-test-slave-n1hw"
server_key: "25b8a6dd5ba9c672abb2cbb5200a6efd"
server_key_iv: "7f2dc4078e70b573be64fd094cee702b"
server_key_version: "encryptionkey@0"
I20260926 16:46:11.557448 6922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:11.579727 6922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:11.580487 6922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:11.580693 6922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:11.581316 6922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:46:11.582401 6922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:46:11.582468 6922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:11.582533 6922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:46:11.582581 6922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:11.591607 6922 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.66:37229
I20260926 16:46:11.591670 7050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.66:37229 every 8 connection(s)
I20260926 16:46:11.592614 6922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
I20260926 16:46:11.602104 6769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:11.594567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:43865 (local address 127.21.12.126:33005)
0926 16:46:11.594710 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:11.594714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:11.595793 (+ 1079us) server_negotiation.cc:407] Connection header received
0926 16:46:11.597313 (+ 1520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:11.597316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:11.597364 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:11.597443 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:11.598408 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:11.598997 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:11.600228 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:11.600471 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:11.600556 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:11.601582 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:11.601585 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:11.601587 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:46:11.601599 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:11.601617 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:46:11.601624 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:11.601685 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:11.601914 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 16:46:11.601966 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 16:46:11.602736 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 6922
I20260926 16:46:11.602854 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance
I20260926 16:46:11.603041 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:11.594989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:33005 (local address 127.21.12.66:43865)
0926 16:46:11.595640 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:11.595681 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:11.597113 (+ 1432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:11.597611 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:11.597617 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:11.597704 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:46:11.598278 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:11.598292 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:11.599118 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:11.599121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:11.600114 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:11.600123 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:11.600612 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:11.600614 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:11.600756 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:11.601329 (+ 573us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:46:11.601331 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:46:11.601371 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:46:11.601438 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:46:11.601745 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:46:11.601772 (+ 27us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:46:11.601777 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 16:46:11.601892 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 16:46:11.602143 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":107,"threads_started":1}
I20260926 16:46:11.603170 21553 external_mini_cluster.cc:1428] Setting key 0f928cf77183ec588198e19f0a2044d7
I20260926 16:46:11.604384 7051 heartbeater.cc:347] Connected to a master server at 127.21.12.126:33005
I20260926 16:46:11.604628 7051 heartbeater.cc:464] Registering TS with master...
I20260926 16:46:11.605243 7051 heartbeater.cc:511] Master 127.21.12.126:33005 requested a full tablet report, sending...
I20260926 16:46:11.605469 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.67:0
--local_ip_for_outbound_sockets=127.21.12.67
--webserver_interface=127.21.12.67
--webserver_port=0
--tserver_master_addrs=127.21.12.126:33005
--builtin_ntp_servers=127.21.12.84:46001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 16:46:11.606349 6730 ts_manager.cc:194] Registered new tserver with Master: 98a67e69247a4df897a1fdf01218a56e (127.21.12.66:37229)
I20260926 16:46:11.607147 6730 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.12.66:43865
W20260926 16:46:11.717011 7054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:11.717293 7054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:11.717332 7054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:11.721447 7054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:11.721513 7054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:11.721572 7054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.67
I20260926 16:46:11.726531 7054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:46001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.67
--webserver_port=0
--tserver_master_addrs=127.21.12.126:33005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.67
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:46:11.727602 7054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:11.728965 7054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:11.739282 7060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:11.739295 7059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:11.739295 7062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:11.739370 7054 server_base.cc:1034] running on GCE node
I20260926 16:46:11.739880 7054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:11.740521 7054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:11.741686 7054 hybrid_clock.cc:648] HybridClock initialized: now 1790441171741670 us; error 26 us; skew 500 ppm
I20260926 16:46:11.743556 7054 webserver.cc:460] Webserver started at http://127.21.12.67:34237/ using document root <none> and password file <none>
I20260926 16:46:11.744100 7054 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:11.744158 7054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:11.744329 7054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:46:11.746413 7054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-2/data/instance:
uuid: "359ed1a512f6431db429d4d13d98c2ed"
format_stamp: "Formatted at 2026-09-26 16:46:11 on dist-test-slave-n1hw"
server_key: "b4c6868a57aeaeb9180e2fda937380df"
server_key_iv: "8fa618f540d2f63113c05f418c69a0fb"
server_key_version: "encryptionkey@0"
I20260926 16:46:11.746950 7054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance:
uuid: "359ed1a512f6431db429d4d13d98c2ed"
format_stamp: "Formatted at 2026-09-26 16:46:11 on dist-test-slave-n1hw"
server_key: "b4c6868a57aeaeb9180e2fda937380df"
server_key_iv: "8fa618f540d2f63113c05f418c69a0fb"
server_key_version: "encryptionkey@0"
I20260926 16:46:11.750634 7054 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:46:11.753198 7068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:11.754611 7054 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 16:46:11.754706 7054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-2/wal
uuid: "359ed1a512f6431db429d4d13d98c2ed"
format_stamp: "Formatted at 2026-09-26 16:46:11 on dist-test-slave-n1hw"
server_key: "b4c6868a57aeaeb9180e2fda937380df"
server_key_iv: "8fa618f540d2f63113c05f418c69a0fb"
server_key_version: "encryptionkey@0"
I20260926 16:46:11.754793 7054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:11.767817 7054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:11.768473 7054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:11.768626 7054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:11.769140 7054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:46:11.770072 7054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:46:11.770119 7054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:11.770159 7054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:46:11.770180 7054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:11.778740 7054 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.67:41993
I20260926 16:46:11.778771 7182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.67:41993 every 8 connection(s)
I20260926 16:46:11.779649 7054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
I20260926 16:46:11.781804 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 7054
I20260926 16:46:11.781920 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance
I20260926 16:46:11.782176 21553 external_mini_cluster.cc:1428] Setting key 9eecaca07d848493322405f0b959aaf5
I20260926 16:46:11.789223 6769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:11.781396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:48049 (local address 127.21.12.126:33005)
0926 16:46:11.781535 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:11.781540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:11.783071 (+ 1531us) server_negotiation.cc:407] Connection header received
0926 16:46:11.784756 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:11.784759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:11.784803 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:11.784870 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:11.785821 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:11.786356 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:11.787441 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:11.787677 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:11.787761 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:11.788788 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:11.788792 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:11.788794 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:46:11.788805 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:11.788825 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:46:11.788855 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:11.788917 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:11.789059 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 16:46:11.789112 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 16:46:11.790100 7185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:11.782135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:33005 (local address 127.21.12.67:48049)
0926 16:46:11.782914 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:11.782954 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:11.784569 (+ 1615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:11.785004 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:11.785011 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:11.785094 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:46:11.785563 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:11.785575 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:11.786464 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:11.786466 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:11.787318 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:11.787327 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:11.787801 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:11.787804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:11.787936 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:11.788541 (+ 605us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:46:11.788543 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:46:11.788581 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:46:11.788639 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:46:11.788929 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:46:11.788935 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:46:11.788939 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 16:46:11.789042 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 16:46:11.789247 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":140,"threads_started":1}
I20260926 16:46:11.791224 7183 heartbeater.cc:347] Connected to a master server at 127.21.12.126:33005
I20260926 16:46:11.791472 7183 heartbeater.cc:464] Registering TS with master...
I20260926 16:46:11.791999 7183 heartbeater.cc:511] Master 127.21.12.126:33005 requested a full tablet report, sending...
I20260926 16:46:11.793002 6730 ts_manager.cc:194] Registered new tserver with Master: 359ed1a512f6431db429d4d13d98c2ed (127.21.12.67:41993)
I20260926 16:46:11.793646 6730 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.12.67:48049
I20260926 16:46:11.796161 21553 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:46:11.801527 6769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:11.798147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49248 (local address 127.21.12.126:33005)
0926 16:46:11.798262 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:11.798265 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:11.798434 (+ 169us) server_negotiation.cc:407] Connection header received
0926 16:46:11.798566 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:11.798568 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:11.798605 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:11.798669 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:11.798884 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:11.799382 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:11.800177 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:11.800379 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:11.800454 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:11.801231 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:11.801233 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:11.801235 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:46:11.801245 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:11.801261 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:46:11.801267 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:11.801316 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:11.801387 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 16:46:11.801432 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 16:46:11.804262 6730 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49248:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:46:11.808862 6730 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:46:11.823518 7194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:11.817837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:46705 (local address 127.0.0.1:39078)
0926 16:46:11.818217 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:11.818270 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:11.818425 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:11.819238 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:11.819245 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:11.819284 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:46:11.819426 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:11.819435 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:11.820617 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:11.820619 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:11.821902 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:11.821910 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:11.822974 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:11.822978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:11.823082 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:11.823121 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 16:46:11.823237 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 16:46:11.823350 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":88,"threads_started":1}
I20260926 16:46:11.823902 7195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:11.817967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39078 (local address 127.21.12.65:46705)
0926 16:46:11.818317 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:11.818332 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:11.818361 (+ 29us) server_negotiation.cc:407] Connection header received
0926 16:46:11.818613 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:11.818620 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:11.818862 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:11.819027 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:46:11.819587 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:11.820509 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:11.822017 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:11.822855 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:11.822966 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:11.823150 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:11.823264 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 16:46:11.823356 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":135,"threads_started":1}
I20260926 16:46:11.824091 7196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:11.818925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:37229 (local address 127.0.0.1:47892)
0926 16:46:11.819547 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:11.819562 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:11.819638 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:11.820228 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:11.820231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:11.820249 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:46:11.820332 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:11.820338 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:11.821892 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:11.821895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:11.822638 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:11.822647 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:11.823812 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:11.823815 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:11.823886 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:11.823903 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 16:46:11.823947 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 16:46:11.823993 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":117,"threads_started":1}
I20260926 16:46:11.824779 7199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:11.819504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47892 (local address 127.21.12.66:37229)
0926 16:46:11.819866 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:11.819874 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:11.819894 (+ 20us) server_negotiation.cc:407] Connection header received
0926 16:46:11.819952 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:11.819956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:11.820102 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:11.820218 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:46:11.820453 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:11.821792 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:11.822757 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:11.823515 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:11.824311 (+ 796us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:11.824493 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:11.824542 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 16:46:11.824629 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":123,"threads_started":1}
I20260926 16:46:11.825120 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:11.818596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.67:41993 (local address 127.0.0.1:42926)
0926 16:46:11.819178 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:11.819189 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:11.819272 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:11.820892 (+ 1620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:11.820896 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:11.820914 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:46:11.820989 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:11.820994 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:11.822503 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:11.822506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:11.823615 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:11.823638 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:11.824738 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:11.824742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:11.824827 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:11.824848 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 16:46:11.824919 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 16:46:11.824989 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":86,"threads_started":1}
I20260926 16:46:11.825222 7198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:11.819380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42926 (local address 127.21.12.67:41993)
0926 16:46:11.820466 (+ 1086us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:11.820476 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:11.820498 (+ 22us) server_negotiation.cc:407] Connection header received
0926 16:46:11.820569 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:11.820574 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:11.820747 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:11.820902 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:46:11.821207 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:11.822379 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:11.823790 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:11.824613 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:11.824725 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:11.824891 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:11.824955 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 16:46:11.825071 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":969,"thread_start_us":155,"threads_started":1}
I20260926 16:46:11.827075 6853 tablet_service.cc:1467] Processing CreateTablet for tablet 88023b83c38548719dc9a5aac5ca4d04 (DEFAULT_TABLE table=test-table [id=cd4939a841fd4e3aa8a75d75cbc22c2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:46:11.827450 6985 tablet_service.cc:1467] Processing CreateTablet for tablet 88023b83c38548719dc9a5aac5ca4d04 (DEFAULT_TABLE table=test-table [id=cd4939a841fd4e3aa8a75d75cbc22c2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:46:11.828068 6853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88023b83c38548719dc9a5aac5ca4d04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:11.828004 7117 tablet_service.cc:1467] Processing CreateTablet for tablet 88023b83c38548719dc9a5aac5ca4d04 (DEFAULT_TABLE table=test-table [id=cd4939a841fd4e3aa8a75d75cbc22c2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:46:11.828480 6985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88023b83c38548719dc9a5aac5ca4d04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:11.828925 7117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88023b83c38548719dc9a5aac5ca4d04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:11.835280 7200 tablet_bootstrap.cc:492] T 88023b83c38548719dc9a5aac5ca4d04 P 5499a3f938df46899e5be26effa512c3: Bootstrap starting.
I20260926 16:46:11.835381 7201 tablet_bootstrap.cc:492] T 88023b83c38548719dc9a5aac5ca4d04 P 359ed1a512f6431db429d4d13d98c2ed: Bootstrap starting.
I20260926 16:46:11.837677 7201 tablet_bootstrap.cc:654] T 88023b83c38548719dc9a5aac5ca4d04 P 359ed1a512f6431db429d4d13d98c2ed: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:11.837677 7200 tablet_bootstrap.cc:654] T 88023b83c38548719dc9a5aac5ca4d04 P 5499a3f938df46899e5be26effa512c3: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:11.838608 7200 log.cc:824] T 88023b83c38548719dc9a5aac5ca4d04 P 5499a3f938df46899e5be26effa512c3: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:11.838608 7201 log.cc:824] T 88023b83c38548719dc9a5aac5ca4d04 P 359ed1a512f6431db429d4d13d98c2ed: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:11.839320 7202 tablet_bootstrap.cc:492] T 88023b83c38548719dc9a5aac5ca4d04 P 98a67e69247a4df897a1fdf01218a56e: Bootstrap starting.
I20260926 16:46:11.840687 7201 tablet_bootstrap.cc:492] T 88023b83c38548719dc9a5aac5ca4d04 P 359ed1a512f6431db429d4d13d98c2ed: No bootstrap required, opened a new log
I20260926 16:46:11.840857 7200 tablet_bootstrap.cc:492] T 88023b83c38548719dc9a5aac5ca4d04 P 5499a3f938df46899e5be26effa512c3: No bootstrap required, opened a new log
I20260926 16:46:11.840878 7201 ts_tablet_manager.cc:1397] T 88023b83c38548719dc9a5aac5ca4d04 P 359ed1a512f6431db429d4d13d98c2ed: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 16:46:11.841035 7200 ts_tablet_manager.cc:1397] T 88023b83c38548719dc9a5aac5ca4d04 P 5499a3f938df46899e5be26effa512c3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 16:46:11.841199 7202 tablet_bootstrap.cc:654] T 88023b83c38548719dc9a5aac5ca4d04 P 98a67e69247a4df897a1fdf01218a56e: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:11.841895 7202 log.cc:824] T 88023b83c38548719dc9a5aac5ca4d04 P 98a67e69247a4df897a1fdf01218a56e: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:11.843451 7202 tablet_bootstrap.cc:492] T 88023b83c38548719dc9a5aac5ca4d04 P 98a67e69247a4df897a1fdf01218a56e: No bootstrap required, opened a new log
I20260926 16:46:11.843634 7202 ts_tablet_manager.cc:1397] T 88023b83c38548719dc9a5aac5ca4d04 P 98a67e69247a4df897a1fdf01218a56e: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260926 16:46:11.845216 7201 raft_consensus.cc:348] T 88023b83c38548719dc9a5aac5ca4d04 P 359ed1a512f6431db429d4d13d98c2ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5499a3f938df46899e5be26effa512c3" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 46705 } } peers { permanent_uuid: "359ed1a512f6431db429d4d13d98c2ed" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 41993 } } peers { permanent_uuid: "98a67e69247a4df897a1fdf01218a56e" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37229 } }
I20260926 16:46:11.845216 7200 raft_consensus.cc:348] T 88023b83c38548719dc9a5aac5ca4d04 P 5499a3f938df46899e5be26effa512c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5499a3f938df46899e5be26effa512c3" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 46705 } } peers { permanent_uuid: "359ed1a512f6431db429d4d13d98c2ed" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 41993 } } peers { permanent_uuid: "98a67e69247a4df897a1fdf01218a56e" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37229 } }
I20260926 16:46:11.845485 7200 raft_consensus.cc:374] T 88023b83c38548719dc9a5aac5ca4d04 P 5499a3f938df46899e5be26effa512c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:46:11.845494 7201 raft_consensus.cc:374] T 88023b83c38548719dc9a5aac5ca4d04 P 359ed1a512f6431db429d4d13d98c2ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:46:11.845546 7201 raft_consensus.cc:729] T 88023b83c38548719dc9a5aac5ca4d04 P 359ed1a512f6431db429d4d13d98c2ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 359ed1a512f6431db429d4d13d98c2ed, State: Initialized, Role: FOLLOWER
I20260926 16:46:11.845546 7200 raft_consensus.cc:729] T 88023b83c38548719dc9a5aac5ca4d04 P 5499a3f938df46899e5be26effa512c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5499a3f938df46899e5be26effa512c3, State: Initialized, Role: FOLLOWER
I20260926 16:46:11.846019 7201 consensus_queue.cc:260] T 88023b83c38548719dc9a5aac5ca4d04 P 359ed1a512f6431db429d4d13d98c2ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5499a3f938df46899e5be26effa512c3" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 46705 } } peers { permanent_uuid: "359ed1a512f6431db429d4d13d98c2ed" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 41993 } } peers { permanent_uuid: "98a67e69247a4df897a1fdf01218a56e" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37229 } }
I20260926 16:46:11.846019 7200 consensus_queue.cc:260] T 88023b83c38548719dc9a5aac5ca4d04 P 5499a3f938df46899e5be26effa512c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5499a3f938df46899e5be26effa512c3" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 46705 } } peers { permanent_uuid: "359ed1a512f6431db429d4d13d98c2ed" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 41993 } } peers { permanent_uuid: "98a67e69247a4df897a1fdf01218a56e" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37229 } }
I20260926 16:46:11.846693 7183 heartbeater.cc:503] Master 127.21.12.126:33005 was elected leader, sending a full tablet report...
I20260926 16:46:11.847021 7201 ts_tablet_manager.cc:1428] T 88023b83c38548719dc9a5aac5ca4d04 P 359ed1a512f6431db429d4d13d98c2ed: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 16:46:11.847024 7200 ts_tablet_manager.cc:1428] T 88023b83c38548719dc9a5aac5ca4d04 P 5499a3f938df46899e5be26effa512c3: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 16:46:11.847232 6919 heartbeater.cc:503] Master 127.21.12.126:33005 was elected leader, sending a full tablet report...
I20260926 16:46:11.849401 7202 raft_consensus.cc:348] T 88023b83c38548719dc9a5aac5ca4d04 P 98a67e69247a4df897a1fdf01218a56e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5499a3f938df46899e5be26effa512c3" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 46705 } } peers { permanent_uuid: "359ed1a512f6431db429d4d13d98c2ed" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 41993 } } peers { permanent_uuid: "98a67e69247a4df897a1fdf01218a56e" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37229 } }
I20260926 16:46:11.849702 7202 raft_consensus.cc:374] T 88023b83c38548719dc9a5aac5ca4d04 P 98a67e69247a4df897a1fdf01218a56e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:46:11.849773 7202 raft_consensus.cc:729] T 88023b83c38548719dc9a5aac5ca4d04 P 98a67e69247a4df897a1fdf01218a56e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98a67e69247a4df897a1fdf01218a56e, State: Initialized, Role: FOLLOWER
I20260926 16:46:11.850309 7202 consensus_queue.cc:260] T 88023b83c38548719dc9a5aac5ca4d04 P 98a67e69247a4df897a1fdf01218a56e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5499a3f938df46899e5be26effa512c3" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 46705 } } peers { permanent_uuid: "359ed1a512f6431db429d4d13d98c2ed" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 41993 } } peers { permanent_uuid: "98a67e69247a4df897a1fdf01218a56e" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37229 } }
I20260926 16:46:11.851389 7202 ts_tablet_manager.cc:1428] T 88023b83c38548719dc9a5aac5ca4d04 P 98a67e69247a4df897a1fdf01218a56e: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260926 16:46:11.851783 7051 heartbeater.cc:503] Master 127.21.12.126:33005 was elected leader, sending a full tablet report...
I20260926 16:46:11.858247 7206 raft_consensus.cc:482] T 88023b83c38548719dc9a5aac5ca4d04 P 359ed1a512f6431db429d4d13d98c2ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:46:11.858407 7206 raft_consensus.cc:504] T 88023b83c38548719dc9a5aac5ca4d04 P 359ed1a512f6431db429d4d13d98c2ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5499a3f938df46899e5be26effa512c3" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 46705 } } peers { permanent_uuid: "359ed1a512f6431db429d4d13d98c2ed" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 41993 } } peers { permanent_uuid: "98a67e69247a4df897a1fdf01218a56e" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37229 } }
I20260926 16:46:11.859409 7206 leader_election.cc:288] T 88023b83c38548719dc9a5aac5ca4d04 P 359ed1a512f6431db429d4d13d98c2ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5499a3f938df46899e5be26effa512c3 (127.21.12.65:46705), 98a67e69247a4df897a1fdf01218a56e (127.21.12.66:37229)
I20260926 16:46:11.863811 7199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:11.859651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:40905 (local address 127.21.12.66:37229)
0926 16:46:11.859768 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:11.859772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:11.859785 (+ 13us) server_negotiation.cc:407] Connection header received
0926 16:46:11.859974 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:11.859979 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:11.860057 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:11.860141 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:46:11.860440 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:11.861239 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:11.862476 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:11.863075 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:11.863205 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:11.863251 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:11.863618 (+ 367us) server_negotiation.cc:299] Negotiation successful
0926 16:46:11.863689 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 16:46:11.863811 7195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:11.859960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:39021 (local address 127.21.12.65:46705)
0926 16:46:11.860072 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:11.860079 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:11.860094 (+ 15us) server_negotiation.cc:407] Connection header received
0926 16:46:11.860142 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:11.860144 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:11.860184 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:11.860254 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:46:11.860623 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:11.861357 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:11.862530 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:11.863114 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:11.863217 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:11.863258 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:11.863618 (+ 360us) server_negotiation.cc:299] Negotiation successful
0926 16:46:11.863689 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 16:46:11.863832 7185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:11.859585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:37229 (local address 127.21.12.67:40905)
0926 16:46:11.859699 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:11.859715 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:11.859798 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:11.860160 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:11.860162 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:11.860176 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:46:11.860326 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:11.860333 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:11.861334 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:11.861337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:11.862354 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:11.862364 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:11.863206 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:11.863209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:11.863380 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:11.863512 (+ 132us) client_negotiation.cc:769] Sending connection context
0926 16:46:11.863593 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 16:46:11.863681 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":101}
I20260926 16:46:11.863849 7209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:11.859585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:46705 (local address 127.21.12.67:39021)
0926 16:46:11.859975 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:11.859986 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:11.860060 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:11.860310 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:11.860313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:11.860326 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:46:11.860444 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:11.860528 (+ 84us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:11.861433 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:11.861436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:11.862428 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:11.862438 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:11.863210 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:11.863213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:11.863400 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:11.863521 (+ 121us) client_negotiation.cc:769] Sending connection context
0926 16:46:11.863593 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 16:46:11.863681 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":284,"thread_start_us":112,"threads_started":1}
I20260926 16:46:11.864492 7005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88023b83c38548719dc9a5aac5ca4d04" candidate_uuid: "359ed1a512f6431db429d4d13d98c2ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98a67e69247a4df897a1fdf01218a56e" is_pre_election: true
I20260926 16:46:11.864665 6873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88023b83c38548719dc9a5aac5ca4d04" candidate_uuid: "359ed1a512f6431db429d4d13d98c2ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5499a3f938df46899e5be26effa512c3" is_pre_election: true
I20260926 16:46:11.864763 7005 raft_consensus.cc:2454] T 88023b83c38548719dc9a5aac5ca4d04 P 98a67e69247a4df897a1fdf01218a56e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 359ed1a512f6431db429d4d13d98c2ed in term 0.
I20260926 16:46:11.864935 6873 raft_consensus.cc:2454] T 88023b83c38548719dc9a5aac5ca4d04 P 5499a3f938df46899e5be26effa512c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 359ed1a512f6431db429d4d13d98c2ed in term 0.
I20260926 16:46:11.865240 7072 leader_election.cc:302] T 88023b83c38548719dc9a5aac5ca4d04 P 359ed1a512f6431db429d4d13d98c2ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 359ed1a512f6431db429d4d13d98c2ed, 98a67e69247a4df897a1fdf01218a56e; no voters:
I20260926 16:46:11.865525 7206 raft_consensus.cc:2781] T 88023b83c38548719dc9a5aac5ca4d04 P 359ed1a512f6431db429d4d13d98c2ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:46:11.865617 7206 raft_consensus.cc:482] T 88023b83c38548719dc9a5aac5ca4d04 P 359ed1a512f6431db429d4d13d98c2ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:46:11.865686 7206 raft_consensus.cc:3037] T 88023b83c38548719dc9a5aac5ca4d04 P 359ed1a512f6431db429d4d13d98c2ed [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:11.866735 7206 raft_consensus.cc:504] T 88023b83c38548719dc9a5aac5ca4d04 P 359ed1a512f6431db429d4d13d98c2ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5499a3f938df46899e5be26effa512c3" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 46705 } } peers { permanent_uuid: "359ed1a512f6431db429d4d13d98c2ed" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 41993 } } peers { permanent_uuid: "98a67e69247a4df897a1fdf01218a56e" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37229 } }
I20260926 16:46:11.867029 7206 leader_election.cc:288] T 88023b83c38548719dc9a5aac5ca4d04 P 359ed1a512f6431db429d4d13d98c2ed [CANDIDATE]: Term 1 election: Requested vote from peers 5499a3f938df46899e5be26effa512c3 (127.21.12.65:46705), 98a67e69247a4df897a1fdf01218a56e (127.21.12.66:37229)
I20260926 16:46:11.867334 7005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88023b83c38548719dc9a5aac5ca4d04" candidate_uuid: "359ed1a512f6431db429d4d13d98c2ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98a67e69247a4df897a1fdf01218a56e"
I20260926 16:46:11.867441 6873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88023b83c38548719dc9a5aac5ca4d04" candidate_uuid: "359ed1a512f6431db429d4d13d98c2ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5499a3f938df46899e5be26effa512c3"
I20260926 16:46:11.867461 7005 raft_consensus.cc:3037] T 88023b83c38548719dc9a5aac5ca4d04 P 98a67e69247a4df897a1fdf01218a56e [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:11.867568 6873 raft_consensus.cc:3037] T 88023b83c38548719dc9a5aac5ca4d04 P 5499a3f938df46899e5be26effa512c3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:11.868537 7005 raft_consensus.cc:2454] T 88023b83c38548719dc9a5aac5ca4d04 P 98a67e69247a4df897a1fdf01218a56e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 359ed1a512f6431db429d4d13d98c2ed in term 1.
I20260926 16:46:11.868541 6873 raft_consensus.cc:2454] T 88023b83c38548719dc9a5aac5ca4d04 P 5499a3f938df46899e5be26effa512c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 359ed1a512f6431db429d4d13d98c2ed in term 1.
I20260926 16:46:11.868824 7069 leader_election.cc:302] T 88023b83c38548719dc9a5aac5ca4d04 P 359ed1a512f6431db429d4d13d98c2ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 359ed1a512f6431db429d4d13d98c2ed, 5499a3f938df46899e5be26effa512c3; no voters:
I20260926 16:46:11.869048 7206 raft_consensus.cc:2781] T 88023b83c38548719dc9a5aac5ca4d04 P 359ed1a512f6431db429d4d13d98c2ed [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:46:11.869261 7206 raft_consensus.cc:686] T 88023b83c38548719dc9a5aac5ca4d04 P 359ed1a512f6431db429d4d13d98c2ed [term 1 LEADER]: Becoming Leader. State: Replica: 359ed1a512f6431db429d4d13d98c2ed, State: Running, Role: LEADER
I20260926 16:46:11.869589 7206 consensus_queue.cc:237] T 88023b83c38548719dc9a5aac5ca4d04 P 359ed1a512f6431db429d4d13d98c2ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5499a3f938df46899e5be26effa512c3" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 46705 } } peers { permanent_uuid: "359ed1a512f6431db429d4d13d98c2ed" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 41993 } } peers { permanent_uuid: "98a67e69247a4df897a1fdf01218a56e" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37229 } }
I20260926 16:46:11.872753 6729 catalog_manager.cc:5495] T 88023b83c38548719dc9a5aac5ca4d04 P 359ed1a512f6431db429d4d13d98c2ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 359ed1a512f6431db429d4d13d98c2ed (127.21.12.67). New cstate: current_term: 1 leader_uuid: "359ed1a512f6431db429d4d13d98c2ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5499a3f938df46899e5be26effa512c3" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 46705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "359ed1a512f6431db429d4d13d98c2ed" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 41993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98a67e69247a4df897a1fdf01218a56e" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37229 } health_report { overall_health: UNKNOWN } } }
I20260926 16:46:11.891791 7198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:11.888455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42932 (local address 127.21.12.67:41993)
0926 16:46:11.888601 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:11.888604 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:11.888617 (+ 13us) server_negotiation.cc:407] Connection header received
0926 16:46:11.888686 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:11.888688 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:11.888739 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:11.888828 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:46:11.889043 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:11.889550 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:11.890434 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:11.890662 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:11.890738 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:11.891000 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:46:11.891418 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:46:11.891538 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:11.891621 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 16:46:11.891666 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 16:46:11.897930 6873 raft_consensus.cc:1261] T 88023b83c38548719dc9a5aac5ca4d04 P 5499a3f938df46899e5be26effa512c3 [term 1 FOLLOWER]: Refusing update from remote peer 359ed1a512f6431db429d4d13d98c2ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:46:11.897930 7005 raft_consensus.cc:1261] T 88023b83c38548719dc9a5aac5ca4d04 P 98a67e69247a4df897a1fdf01218a56e [term 1 FOLLOWER]: Refusing update from remote peer 359ed1a512f6431db429d4d13d98c2ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260926 16:46:11.898193 6920 tablet.cc:2369] T 88023b83c38548719dc9a5aac5ca4d04 P 5499a3f938df46899e5be26effa512c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:46:11.898697 7206 consensus_queue.cc:1035] T 88023b83c38548719dc9a5aac5ca4d04 P 359ed1a512f6431db429d4d13d98c2ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "98a67e69247a4df897a1fdf01218a56e" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:46:11.898933 7210 consensus_queue.cc:1035] T 88023b83c38548719dc9a5aac5ca4d04 P 359ed1a512f6431db429d4d13d98c2ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "5499a3f938df46899e5be26effa512c3" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 46705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:46:11.917248 6730 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:46:11.917405 6730 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:46:11.919764 6730 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0469d4f67666489e9581eeb0d0cff14d: Sending DeleteTablet for 3 replicas of tablet 88023b83c38548719dc9a5aac5ca4d04
I20260926 16:46:11.920593 6853 tablet_service.cc:1514] Processing DeleteTablet for tablet 88023b83c38548719dc9a5aac5ca4d04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:46:11 UTC) from {username='slave'} at 127.0.0.1:39078
I20260926 16:46:11.920595 6985 tablet_service.cc:1514] Processing DeleteTablet for tablet 88023b83c38548719dc9a5aac5ca4d04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:46:11 UTC) from {username='slave'} at 127.0.0.1:47892
I20260926 16:46:11.920629 7117 tablet_service.cc:1514] Processing DeleteTablet for tablet 88023b83c38548719dc9a5aac5ca4d04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:46:11 UTC) from {username='slave'} at 127.0.0.1:42926
I20260926 16:46:11.921087 7222 tablet_replica.cc:330] T 88023b83c38548719dc9a5aac5ca4d04 P 5499a3f938df46899e5be26effa512c3: stopping tablet replica
I20260926 16:46:11.921087 7223 tablet_replica.cc:330] T 88023b83c38548719dc9a5aac5ca4d04 P 98a67e69247a4df897a1fdf01218a56e: stopping tablet replica
I20260926 16:46:11.921394 7223 raft_consensus.cc:2229] T 88023b83c38548719dc9a5aac5ca4d04 P 98a67e69247a4df897a1fdf01218a56e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:46:11.921394 7222 raft_consensus.cc:2229] T 88023b83c38548719dc9a5aac5ca4d04 P 5499a3f938df46899e5be26effa512c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:46:11.921700 7223 raft_consensus.cc:2258] T 88023b83c38548719dc9a5aac5ca4d04 P 98a67e69247a4df897a1fdf01218a56e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:46:11.921700 7222 raft_consensus.cc:2258] T 88023b83c38548719dc9a5aac5ca4d04 P 5499a3f938df46899e5be26effa512c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:46:11.921986 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 6790
I20260926 16:46:11.925050 7224 tablet_replica.cc:330] T 88023b83c38548719dc9a5aac5ca4d04 P 359ed1a512f6431db429d4d13d98c2ed: stopping tablet replica
I20260926 16:46:11.925222 7223 ts_tablet_manager.cc:1905] T 88023b83c38548719dc9a5aac5ca4d04 P 98a67e69247a4df897a1fdf01218a56e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:46:11.925477 7224 raft_consensus.cc:2229] T 88023b83c38548719dc9a5aac5ca4d04 P 359ed1a512f6431db429d4d13d98c2ed [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:46:11.925990 7224 raft_consensus.cc:2258] T 88023b83c38548719dc9a5aac5ca4d04 P 359ed1a512f6431db429d4d13d98c2ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:46:11.927567 7224 ts_tablet_manager.cc:1905] T 88023b83c38548719dc9a5aac5ca4d04 P 359ed1a512f6431db429d4d13d98c2ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:46:11.928146 7223 ts_tablet_manager.cc:1918] T 88023b83c38548719dc9a5aac5ca4d04 P 98a67e69247a4df897a1fdf01218a56e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:46:11.928251 7223 log.cc:1196] T 88023b83c38548719dc9a5aac5ca4d04 P 98a67e69247a4df897a1fdf01218a56e: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-1/wal/wals/88023b83c38548719dc9a5aac5ca4d04
I20260926 16:46:11.928556 7223 ts_tablet_manager.cc:1939] T 88023b83c38548719dc9a5aac5ca4d04 P 98a67e69247a4df897a1fdf01218a56e: Deleting consensus metadata
W20260926 16:46:11.929282 6714 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.12.65:46705 (error 108)
I20260926 16:46:11.929876 6717 catalog_manager.cc:4841] TS 98a67e69247a4df897a1fdf01218a56e (127.21.12.66:37229): tablet 88023b83c38548719dc9a5aac5ca4d04 (table test-table [id=cd4939a841fd4e3aa8a75d75cbc22c2f]) successfully deleted
I20260926 16:46:11.930063 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 6922
I20260926 16:46:11.932657 7224 ts_tablet_manager.cc:1918] T 88023b83c38548719dc9a5aac5ca4d04 P 359ed1a512f6431db429d4d13d98c2ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:46:11.932771 7224 log.cc:1196] T 88023b83c38548719dc9a5aac5ca4d04 P 359ed1a512f6431db429d4d13d98c2ed: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440924937351-21553-0/minicluster-data/ts-2/wal/wals/88023b83c38548719dc9a5aac5ca4d04
I20260926 16:46:11.933053 7224 ts_tablet_manager.cc:1939] T 88023b83c38548719dc9a5aac5ca4d04 P 359ed1a512f6431db429d4d13d98c2ed: Deleting consensus metadata
I20260926 16:46:11.934110 6714 catalog_manager.cc:4841] TS 359ed1a512f6431db429d4d13d98c2ed (127.21.12.67:41993): tablet 88023b83c38548719dc9a5aac5ca4d04 (table test-table [id=cd4939a841fd4e3aa8a75d75cbc22c2f]) successfully deleted
I20260926 16:46:11.934741 7196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:11.934412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:46705 (local address 127.0.0.1:39088)
0926 16:46:11.934550 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:11.934631 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.65:46705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 16:46:11.935186 6714 catalog_manager.cc:4584] TS 5499a3f938df46899e5be26effa512c3 (127.21.12.65:46705): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 88023b83c38548719dc9a5aac5ca4d04: Network error: Client connection negotiation failed: client connection to 127.21.12.65:46705: connect: Connection refused (error 111)
I20260926 16:46:11.936959 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 7054
I20260926 16:46:11.942337 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 6699
2026-09-26T16:46:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1009 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:46:12 dist-test-slave-n1hw krb5kdc[7228](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:46:12 dist-test-slave-n1hw krb5kdc[7228](info): set up 2 sockets
Sep 26 16:46:12 dist-test-slave-n1hw krb5kdc[7228](info): commencing operation
krb5kdc: starting...
W20260926 16:46:12.979879 21553 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Sep 26 16:46:13 dist-test-slave-n1hw krb5kdc[7228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:46:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:46:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
I20260926 16:46:13.104442 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:43481
--webserver_interface=127.21.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:36395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:43481
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:46:13.212782 7241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:13.213085 7241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:13.213153 7241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:13.217206 7241 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:46:13.217300 7241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:13.217339 7241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:13.217394 7241 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:46:13.217430 7241 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:46:13.222654 7241 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:36395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:43481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:43481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:46:13.223747 7241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:13.224785 7241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:13.233332 7248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:13.233379 7250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:13.233371 7241 server_base.cc:1034] running on GCE node
W20260926 16:46:13.233408 7246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:13.234464 7241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:13.235599 7241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:13.236760 7241 hybrid_clock.cc:648] HybridClock initialized: now 1790441173236746 us; error 27 us; skew 500 ppm
Sep 26 16:46:13 dist-test-slave-n1hw krb5kdc[7228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441173, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:46:13.243363 7241 init.cc:377] Logged in from keytab as kudu/127.21.12.126@KRBTEST.COM (short username kudu)
I20260926 16:46:13.244323 7241 webserver.cc:460] Webserver started at http://127.21.12.126:34245/ using document root <none> and password file <none>
I20260926 16:46:13.244882 7241 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:13.244969 7241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:13.245203 7241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:46:13.247323 7241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "ccdb698b9b9648cd853fcc6ff43f626c"
format_stamp: "Formatted at 2026-09-26 16:46:13 on dist-test-slave-n1hw"
server_key: "75da0f04ed95fa6b0f21d1bd75100a8c"
server_key_iv: "0aec05ffed29186203a414ce052ffc70"
server_key_version: "encryptionkey@0"
I20260926 16:46:13.247892 7241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "ccdb698b9b9648cd853fcc6ff43f626c"
format_stamp: "Formatted at 2026-09-26 16:46:13 on dist-test-slave-n1hw"
server_key: "75da0f04ed95fa6b0f21d1bd75100a8c"
server_key_iv: "0aec05ffed29186203a414ce052ffc70"
server_key_version: "encryptionkey@0"
I20260926 16:46:13.251695 7241 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 16:46:13.254212 7257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:13.255586 7241 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:46:13.255744 7241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "ccdb698b9b9648cd853fcc6ff43f626c"
format_stamp: "Formatted at 2026-09-26 16:46:13 on dist-test-slave-n1hw"
server_key: "75da0f04ed95fa6b0f21d1bd75100a8c"
server_key_iv: "0aec05ffed29186203a414ce052ffc70"
server_key_version: "encryptionkey@0"
I20260926 16:46:13.255860 7241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:13.282565 7241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:13.285624 7241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:13.285846 7241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:13.292982 7241 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.126:43481
I20260926 16:46:13.293017 7309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.126:43481 every 8 connection(s)
I20260926 16:46:13.293938 7241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:46:13.296674 7310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:13.300938 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 7241
I20260926 16:46:13.301072 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:46:13.301363 21553 external_mini_cluster.cc:1428] Setting key 5ff0252ec7bfd041250bfb975f3a20a6
I20260926 16:46:13.302552 7310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccdb698b9b9648cd853fcc6ff43f626c: Bootstrap starting.
I20260926 16:46:13.304824 7310 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccdb698b9b9648cd853fcc6ff43f626c: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:13.305549 7310 log.cc:824] T 00000000000000000000000000000000 P ccdb698b9b9648cd853fcc6ff43f626c: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:13.307544 7310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccdb698b9b9648cd853fcc6ff43f626c: No bootstrap required, opened a new log
Sep 26 16:46:13 dist-test-slave-n1hw krb5kdc[7228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:46:13.311931 7310 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccdb698b9b9648cd853fcc6ff43f626c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccdb698b9b9648cd853fcc6ff43f626c" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 43481 } }
I20260926 16:46:13.312233 7310 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccdb698b9b9648cd853fcc6ff43f626c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:46:13.312288 7310 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccdb698b9b9648cd853fcc6ff43f626c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccdb698b9b9648cd853fcc6ff43f626c, State: Initialized, Role: FOLLOWER
I20260926 16:46:13.312796 7310 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccdb698b9b9648cd853fcc6ff43f626c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccdb698b9b9648cd853fcc6ff43f626c" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 43481 } }
I20260926 16:46:13.312971 7310 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccdb698b9b9648cd853fcc6ff43f626c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:46:13.313055 7310 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccdb698b9b9648cd853fcc6ff43f626c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:46:13.313153 7310 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccdb698b9b9648cd853fcc6ff43f626c [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:13.314304 7310 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccdb698b9b9648cd853fcc6ff43f626c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccdb698b9b9648cd853fcc6ff43f626c" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 43481 } }
I20260926 16:46:13.314716 7310 leader_election.cc:302] T 00000000000000000000000000000000 P ccdb698b9b9648cd853fcc6ff43f626c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccdb698b9b9648cd853fcc6ff43f626c; no voters:
I20260926 16:46:13.315033 7310 leader_election.cc:288] T 00000000000000000000000000000000 P ccdb698b9b9648cd853fcc6ff43f626c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:46:13.315158 7315 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccdb698b9b9648cd853fcc6ff43f626c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:46:13.315392 7315 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccdb698b9b9648cd853fcc6ff43f626c [term 1 LEADER]: Becoming Leader. State: Replica: ccdb698b9b9648cd853fcc6ff43f626c, State: Running, Role: LEADER
I20260926 16:46:13.315719 7315 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccdb698b9b9648cd853fcc6ff43f626c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccdb698b9b9648cd853fcc6ff43f626c" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 43481 } }
I20260926 16:46:13.316394 7310 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccdb698b9b9648cd853fcc6ff43f626c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:46:13.317592 7313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:13.302876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37194 (local address 127.21.12.126:43481)
0926 16:46:13.303311 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:13.303327 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:13.303361 (+ 34us) server_negotiation.cc:407] Connection header received
0926 16:46:13.304041 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:13.304059 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:13.304387 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:13.304723 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:13.305574 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:13.306832 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:13.307718 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:13.308386 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:13.308558 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:13.311519 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:13.311525 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:13.311541 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:13.311571 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:13.313741 (+ 2170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:13.314183 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:13.314192 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:13.314199 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:13.314346 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:13.314654 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:13.314657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:13.314659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:13.315188 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:13.316126 (+ 938us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:13.316271 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 16:46:13.316572 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":135,"threads_started":1}
I20260926 16:46:13.317597 7317 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccdb698b9b9648cd853fcc6ff43f626c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccdb698b9b9648cd853fcc6ff43f626c. Latest consensus state: current_term: 1 leader_uuid: "ccdb698b9b9648cd853fcc6ff43f626c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccdb698b9b9648cd853fcc6ff43f626c" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 43481 } } }
I20260926 16:46:13.317786 7317 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccdb698b9b9648cd853fcc6ff43f626c [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:13.317612 7316 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccdb698b9b9648cd853fcc6ff43f626c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccdb698b9b9648cd853fcc6ff43f626c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccdb698b9b9648cd853fcc6ff43f626c" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 43481 } } }
I20260926 16:46:13.318099 7316 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccdb698b9b9648cd853fcc6ff43f626c [sys.catalog]: This master's current role is: LEADER
W20260926 16:46:13.320937 7330 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ccdb698b9b9648cd853fcc6ff43f626c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 16:46:13.321035 7330 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 16:46:13.321110 7324 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:46:13.321857 7324 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:46:13.326880 7324 catalog_manager.cc:1347] Generated new cluster ID: 76f3dbb647194ff9b152e31bed05828d
I20260926 16:46:13.326946 7324 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:46:13.335708 7324 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:46:13.336982 7324 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:46:13.351279 7324 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccdb698b9b9648cd853fcc6ff43f626c: Generated new TSK 0
I20260926 16:46:13.352146 7324 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab.
Entry for principal kudu/127.21.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab.
I20260926 16:46:13.380841 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.65:0
--local_ip_for_outbound_sockets=127.21.12.65
--webserver_interface=127.21.12.65
--webserver_port=0
--tserver_master_addrs=127.21.12.126:43481
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab
--principal=kudu/127.21.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:36395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:46:13.490386 7336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:13.490669 7336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:13.490739 7336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:13.494781 7336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:13.494866 7336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:13.494954 7336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.65
I20260926 16:46:13.500151 7336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:36395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab
--principal=kudu/127.21.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.65
--webserver_port=0
--tserver_master_addrs=127.21.12.126:43481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.65
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:46:13.501302 7336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:13.502405 7336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:13.511870 7342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:13.511889 7344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:13.511905 7341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:13.511938 7336 server_base.cc:1034] running on GCE node
I20260926 16:46:13.512444 7336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:13.513065 7336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:13.514273 7336 hybrid_clock.cc:648] HybridClock initialized: now 1790441173514254 us; error 29 us; skew 500 ppm
Sep 26 16:46:13 dist-test-slave-n1hw krb5kdc[7228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441173, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:46:13.517078 7336 init.cc:377] Logged in from keytab as kudu/127.21.12.65@KRBTEST.COM (short username kudu)
I20260926 16:46:13.518028 7336 webserver.cc:460] Webserver started at http://127.21.12.65:36903/ using document root <none> and password file <none>
I20260926 16:46:13.518589 7336 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:13.518674 7336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:13.518889 7336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:46:13.520859 7336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-0/data/instance:
uuid: "05f6c54cd0eb4dbca98007b1dbd409dd"
format_stamp: "Formatted at 2026-09-26 16:46:13 on dist-test-slave-n1hw"
server_key: "4b7539f5d25d9e73b072b3a7c3d44c63"
server_key_iv: "fc6617f94c2d6fb2bd482c9be8623a81"
server_key_version: "encryptionkey@0"
I20260926 16:46:13.521443 7336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance:
uuid: "05f6c54cd0eb4dbca98007b1dbd409dd"
format_stamp: "Formatted at 2026-09-26 16:46:13 on dist-test-slave-n1hw"
server_key: "4b7539f5d25d9e73b072b3a7c3d44c63"
server_key_iv: "fc6617f94c2d6fb2bd482c9be8623a81"
server_key_version: "encryptionkey@0"
I20260926 16:46:13.525056 7336 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 16:46:13.527491 7351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:13.528671 7336 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 16:46:13.528815 7336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-0/wal
uuid: "05f6c54cd0eb4dbca98007b1dbd409dd"
format_stamp: "Formatted at 2026-09-26 16:46:13 on dist-test-slave-n1hw"
server_key: "4b7539f5d25d9e73b072b3a7c3d44c63"
server_key_iv: "fc6617f94c2d6fb2bd482c9be8623a81"
server_key_version: "encryptionkey@0"
I20260926 16:46:13.528927 7336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:13.540632 7336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:13.543824 7336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:13.544034 7336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:13.544601 7336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:46:13.545392 7336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:46:13.545476 7336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:13.545563 7336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:46:13.545612 7336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:13.554443 7336 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.65:33915
I20260926 16:46:13.554462 7466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.65:33915 every 8 connection(s)
I20260926 16:46:13.555356 7336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
I20260926 16:46:13.557221 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 7336
I20260926 16:46:13.557350 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance
I20260926 16:46:13.557608 21553 external_mini_cluster.cc:1428] Setting key 615f13dff877b4599a58998de9fe6649
Sep 26 16:46:13 dist-test-slave-n1hw krb5kdc[7228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441173, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.12.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.66@KRBTEST.COM" created.
I20260926 16:46:13.570720 7313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:13.557144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:44211 (local address 127.21.12.126:43481)
0926 16:46:13.557311 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:13.557318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:13.558232 (+ 914us) server_negotiation.cc:407] Connection header received
0926 16:46:13.559961 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:13.559963 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:13.560004 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:13.560101 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:13.561408 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:13.562302 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:13.563488 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:13.563928 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:13.564070 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:13.566944 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:13.566947 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:13.566950 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:13.566971 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:13.568477 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:13.568965 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:13.568971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:13.568976 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:13.569070 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:13.569384 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:13.569386 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:13.569387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:13.569588 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:13.569666 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:13.570300 (+ 634us) server_negotiation.cc:299] Negotiation successful
0926 16:46:13.570459 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 16:46:13.571449 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:13.557547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43481 (local address 127.21.12.65:44211)
0926 16:46:13.558084 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:13.558126 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:13.559728 (+ 1602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:13.560420 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:13.560429 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:13.560791 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:46:13.561264 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:13.561277 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:13.562426 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:13.562429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:13.563372 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:13.563383 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:13.564146 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:13.564149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:13.564301 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:13.564943 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:46:13.564959 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:46:13.566770 (+ 1811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:46:13.568585 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:13.568590 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:13.568601 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:13.568852 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:13.569157 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:13.569159 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:13.569161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:13.569296 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:13.569715 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:46:13.569721 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:46:13.570073 (+ 352us) client_negotiation.cc:769] Sending connection context
0926 16:46:13.570239 (+ 166us) client_negotiation.cc:240] Negotiation successful
0926 16:46:13.570540 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":150,"threads_started":1}
I20260926 16:46:13.572846 7467 heartbeater.cc:347] Connected to a master server at 127.21.12.126:43481
I20260926 16:46:13.573101 7467 heartbeater.cc:464] Registering TS with master...
I20260926 16:46:13.573576 7467 heartbeater.cc:511] Master 127.21.12.126:43481 requested a full tablet report, sending...
I20260926 16:46:13.575086 7274 ts_manager.cc:194] Registered new tserver with Master: 05f6c54cd0eb4dbca98007b1dbd409dd (127.21.12.65:33915)
I20260926 16:46:13.576157 7274 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.65@KRBTEST.COM'} at 127.21.12.65:44211
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab.
Entry for principal kudu/127.21.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab.
I20260926 16:46:13.588461 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.66:0
--local_ip_for_outbound_sockets=127.21.12.66
--webserver_interface=127.21.12.66
--webserver_port=0
--tserver_master_addrs=127.21.12.126:43481
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab
--principal=kudu/127.21.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:36395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:46:13.699965 7472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:13.700256 7472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:13.700331 7472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:13.704572 7472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:13.704658 7472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:13.704746 7472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.66
I20260926 16:46:13.710266 7472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:36395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab
--principal=kudu/127.21.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.66
--webserver_port=0
--tserver_master_addrs=127.21.12.126:43481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.66
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:46:13.711330 7472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:13.712334 7472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:13.722028 7480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:13.722028 7477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:13.722177 7472 server_base.cc:1034] running on GCE node
W20260926 16:46:13.722075 7478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:13.722873 7472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:13.723506 7472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:13.724713 7472 hybrid_clock.cc:648] HybridClock initialized: now 1790441173724696 us; error 31 us; skew 500 ppm
Sep 26 16:46:13 dist-test-slave-n1hw krb5kdc[7228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441173, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:46:13.727384 7472 init.cc:377] Logged in from keytab as kudu/127.21.12.66@KRBTEST.COM (short username kudu)
I20260926 16:46:13.728200 7472 webserver.cc:460] Webserver started at http://127.21.12.66:42391/ using document root <none> and password file <none>
I20260926 16:46:13.728704 7472 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:13.728763 7472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:13.728920 7472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:46:13.730888 7472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-1/data/instance:
uuid: "7df6d6270bf041e59390794d64e9ccea"
format_stamp: "Formatted at 2026-09-26 16:46:13 on dist-test-slave-n1hw"
server_key: "b9c3ec30655d6bf8ea2d07575f15565d"
server_key_iv: "c07206509658abc6ee3456464dae9b4d"
server_key_version: "encryptionkey@0"
I20260926 16:46:13.731439 7472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance:
uuid: "7df6d6270bf041e59390794d64e9ccea"
format_stamp: "Formatted at 2026-09-26 16:46:13 on dist-test-slave-n1hw"
server_key: "b9c3ec30655d6bf8ea2d07575f15565d"
server_key_iv: "c07206509658abc6ee3456464dae9b4d"
server_key_version: "encryptionkey@0"
I20260926 16:46:13.735224 7472 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 16:46:13.737491 7487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:13.738831 7472 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:46:13.738950 7472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-1/wal
uuid: "7df6d6270bf041e59390794d64e9ccea"
format_stamp: "Formatted at 2026-09-26 16:46:13 on dist-test-slave-n1hw"
server_key: "b9c3ec30655d6bf8ea2d07575f15565d"
server_key_iv: "c07206509658abc6ee3456464dae9b4d"
server_key_version: "encryptionkey@0"
I20260926 16:46:13.739058 7472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:13.749473 7472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:13.752622 7472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:13.752872 7472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:13.753458 7472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:46:13.754341 7472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:46:13.754388 7472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:13.754436 7472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:46:13.754458 7472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:13.763115 7472 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.66:37803
I20260926 16:46:13.763190 7600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.66:37803 every 8 connection(s)
I20260926 16:46:13.764002 7472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
I20260926 16:46:13.764653 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 7472
I20260926 16:46:13.764784 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance
I20260926 16:46:13.765053 21553 external_mini_cluster.cc:1428] Setting key 93e9c61a4f7741d2c0072d7d753f7c77
Sep 26 16:46:13 dist-test-slave-n1hw krb5kdc[7228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441173, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.66@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.12.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.67@KRBTEST.COM" created.
I20260926 16:46:13.778039 7313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:13.765865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:50533 (local address 127.21.12.126:43481)
0926 16:46:13.765997 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:13.766001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:13.767008 (+ 1007us) server_negotiation.cc:407] Connection header received
0926 16:46:13.768516 (+ 1508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:13.768519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:13.768574 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:13.768670 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:13.769839 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:13.770368 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:13.771503 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:13.771730 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:13.771838 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:13.774289 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:13.774293 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:13.774301 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:13.774323 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:13.775732 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:13.776263 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:13.776266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:13.776293 (+ 27us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:13.776395 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:13.776778 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:13.776785 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:13.776790 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:13.777043 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:13.777152 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:13.777630 (+ 478us) server_negotiation.cc:299] Negotiation successful
0926 16:46:13.777807 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 16:46:13.778744 7603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:13.766403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43481 (local address 127.21.12.66:50533)
0926 16:46:13.766871 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:13.766911 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:13.768246 (+ 1335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:13.768845 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:13.768851 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:13.769207 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:46:13.769676 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:13.769689 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:13.770488 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:13.770492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:13.771378 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:13.771387 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:13.771875 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:13.771879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:13.772051 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:13.772648 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:46:13.772667 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:46:13.774130 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:46:13.775843 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:13.775848 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:13.775863 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:13.776126 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:13.776510 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:13.776516 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:13.776521 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:13.776673 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:13.777158 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:46:13.777163 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:46:13.777385 (+ 222us) client_negotiation.cc:769] Sending connection context
0926 16:46:13.777532 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 16:46:13.777828 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":107,"threads_started":1}
I20260926 16:46:13.779958 7601 heartbeater.cc:347] Connected to a master server at 127.21.12.126:43481
I20260926 16:46:13.780212 7601 heartbeater.cc:464] Registering TS with master...
I20260926 16:46:13.780819 7601 heartbeater.cc:511] Master 127.21.12.126:43481 requested a full tablet report, sending...
I20260926 16:46:13.782002 7274 ts_manager.cc:194] Registered new tserver with Master: 7df6d6270bf041e59390794d64e9ccea (127.21.12.66:37803)
I20260926 16:46:13.782567 7274 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.66@KRBTEST.COM'} at 127.21.12.66:50533
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab.
Entry for principal kudu/127.21.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab.
I20260926 16:46:13.793102 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.67:0
--local_ip_for_outbound_sockets=127.21.12.67
--webserver_interface=127.21.12.67
--webserver_port=0
--tserver_master_addrs=127.21.12.126:43481
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab
--principal=kudu/127.21.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:36395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:46:13.902575 7607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:13.902872 7607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:13.902947 7607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:13.906913 7607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:13.906996 7607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:13.907083 7607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.67
I20260926 16:46:13.912192 7607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:36395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab
--principal=kudu/127.21.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.67
--webserver_port=0
--tserver_master_addrs=127.21.12.126:43481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.67
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:46:13.913219 7607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:13.914249 7607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:13.924263 7612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:13.924263 7615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:13.924278 7613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:13.924382 7607 server_base.cc:1034] running on GCE node
I20260926 16:46:13.924882 7607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:13.925529 7607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:13.926710 7607 hybrid_clock.cc:648] HybridClock initialized: now 1790441173926674 us; error 48 us; skew 500 ppm
Sep 26 16:46:13 dist-test-slave-n1hw krb5kdc[7228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441173, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:46:13.929405 7607 init.cc:377] Logged in from keytab as kudu/127.21.12.67@KRBTEST.COM (short username kudu)
I20260926 16:46:13.930332 7607 webserver.cc:460] Webserver started at http://127.21.12.67:45997/ using document root <none> and password file <none>
I20260926 16:46:13.930879 7607 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:13.930960 7607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:13.931198 7607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:46:13.933354 7607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-2/data/instance:
uuid: "6201cadefff4480ea94cf0537bb3f328"
format_stamp: "Formatted at 2026-09-26 16:46:13 on dist-test-slave-n1hw"
server_key: "394b9f33cbbf0169c4d760f686fe3d23"
server_key_iv: "434ec5309bc381d3f65853cdcc116fe4"
server_key_version: "encryptionkey@0"
I20260926 16:46:13.934026 7607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance:
uuid: "6201cadefff4480ea94cf0537bb3f328"
format_stamp: "Formatted at 2026-09-26 16:46:13 on dist-test-slave-n1hw"
server_key: "394b9f33cbbf0169c4d760f686fe3d23"
server_key_iv: "434ec5309bc381d3f65853cdcc116fe4"
server_key_version: "encryptionkey@0"
I20260926 16:46:13.937614 7607 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:46:13.940129 7622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:13.941378 7607 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:46:13.941512 7607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-2/wal
uuid: "6201cadefff4480ea94cf0537bb3f328"
format_stamp: "Formatted at 2026-09-26 16:46:13 on dist-test-slave-n1hw"
server_key: "394b9f33cbbf0169c4d760f686fe3d23"
server_key_iv: "434ec5309bc381d3f65853cdcc116fe4"
server_key_version: "encryptionkey@0"
I20260926 16:46:13.941630 7607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:13.962044 7607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:13.965474 7607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:13.965696 7607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:13.966326 7607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:46:13.967187 7607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:46:13.967255 7607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:13.967324 7607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:46:13.967363 7607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:13.976672 7607 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.67:42741
I20260926 16:46:13.976691 7737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.67:42741 every 8 connection(s)
I20260926 16:46:13.977635 7607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
I20260926 16:46:13.979318 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 7607
I20260926 16:46:13.979439 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance
I20260926 16:46:13.979768 21553 external_mini_cluster.cc:1428] Setting key 1361b519e1952b43eefd4adcacd41709
Sep 26 16:46:13 dist-test-slave-n1hw krb5kdc[7228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441173, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.67@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:46:13.990936 7313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:13.979495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:55029 (local address 127.21.12.126:43481)
0926 16:46:13.979648 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:13.979653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:13.980684 (+ 1031us) server_negotiation.cc:407] Connection header received
0926 16:46:13.982327 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:13.982330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:13.982374 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:13.982443 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:13.983664 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:13.984185 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:13.985270 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:13.985482 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:13.985562 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:13.987942 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:13.987945 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:13.987947 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:13.987971 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:13.989202 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:13.989651 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:13.989654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:13.989655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:13.989728 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:13.990052 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:13.990053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:13.990054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:13.990217 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:13.990290 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:13.990722 (+ 432us) server_negotiation.cc:299] Negotiation successful
0926 16:46:13.990816 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 16:46:13.991781 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:13.979966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:43481 (local address 127.21.12.67:55029)
0926 16:46:13.980527 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:13.980593 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:13.982145 (+ 1552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:13.982603 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:13.982610 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:13.982959 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:46:13.983539 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:13.983552 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:13.984295 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:13.984298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:13.985168 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:13.985176 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:13.985611 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:13.985614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:13.985801 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:13.986373 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:46:13.986389 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:46:13.987793 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:46:13.989302 (+ 1509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:13.989307 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:13.989318 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:13.989568 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:13.989844 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:13.989846 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:13.989847 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:13.989976 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:13.990298 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:46:13.990302 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:46:13.990517 (+ 215us) client_negotiation.cc:769] Sending connection context
0926 16:46:13.990680 (+ 163us) client_negotiation.cc:240] Negotiation successful
0926 16:46:13.990919 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":138,"threads_started":1}
I20260926 16:46:13.992949 7738 heartbeater.cc:347] Connected to a master server at 127.21.12.126:43481
I20260926 16:46:13.993196 7738 heartbeater.cc:464] Registering TS with master...
I20260926 16:46:13.993666 7738 heartbeater.cc:511] Master 127.21.12.126:43481 requested a full tablet report, sending...
I20260926 16:46:13.994724 7271 ts_manager.cc:194] Registered new tserver with Master: 6201cadefff4480ea94cf0537bb3f328 (127.21.12.67:42741)
I20260926 16:46:13.995322 7271 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.67@KRBTEST.COM'} at 127.21.12.67:55029
I20260926 16:46:14.005169 21553 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:46:14.014890 7313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:14.007141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37206 (local address 127.21.12.126:43481)
0926 16:46:14.007254 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:14.007257 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:14.007453 (+ 196us) server_negotiation.cc:407] Connection header received
0926 16:46:14.007521 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:14.007523 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:14.007568 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:14.007638 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:14.008244 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:14.008772 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:14.009593 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:14.009832 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:14.009921 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:14.011631 (+ 1710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:14.011634 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:14.011636 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:14.011656 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:14.012980 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:14.013487 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:14.013489 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:14.013490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:14.013588 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:14.013944 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:14.013946 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:14.013948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:14.014144 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:14.014272 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:14.014641 (+ 369us) server_negotiation.cc:299] Negotiation successful
0926 16:46:14.014722 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 16:46:14.018357 7271 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37206:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:46:14.021339 7271 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:46:14.035826 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:14.029672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.67:42741 (local address 127.0.0.1:46368)
0926 16:46:14.030789 (+ 1117us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:14.030843 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:14.031036 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:14.031831 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:14.031837 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:14.031859 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:46:14.032074 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:14.032081 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:14.033379 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:14.033382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:14.034481 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:14.034491 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:14.035402 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:14.035404 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:14.035484 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:14.035515 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 16:46:14.035611 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 16:46:14.035708 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":949,"mutex_wait_us":68,"thread_start_us":137,"threads_started":1}
I20260926 16:46:14.035923 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:14.029810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46368 (local address 127.21.12.67:42741)
0926 16:46:14.030779 (+ 969us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:14.030789 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:14.030926 (+ 137us) server_negotiation.cc:407] Connection header received
0926 16:46:14.031491 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:14.031496 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:14.031670 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:14.031841 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:46:14.032241 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:14.033048 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:14.034604 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:14.035292 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:14.035406 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:14.035550 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:14.035658 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 16:46:14.035766 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":880,"thread_start_us":98,"threads_started":1}
I20260926 16:46:14.037504 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:14.030133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44070 (local address 127.21.12.66:37803)
0926 16:46:14.030552 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:14.030564 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:14.031000 (+ 436us) server_negotiation.cc:407] Connection header received
0926 16:46:14.031359 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:14.031365 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:14.031577 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:14.031746 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:46:14.032224 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:14.033414 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:14.035078 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:14.036144 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:14.036266 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:14.036422 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:14.037229 (+ 807us) server_negotiation.cc:299] Negotiation successful
0926 16:46:14.037331 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":100,"threads_started":1}
I20260926 16:46:14.037855 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:14.029688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:37803 (local address 127.0.0.1:44070)
0926 16:46:14.030915 (+ 1227us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:14.030927 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:14.031068 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:14.031907 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:14.031910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:14.031943 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:46:14.032102 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:14.032109 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:14.033570 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:14.033573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:14.034938 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:14.034949 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:14.036262 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:14.036266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:14.036923 (+ 657us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:14.036955 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 16:46:14.037025 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 16:46:14.037096 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1156,"mutex_wait_us":110,"spinlock_wait_cycles":4608,"thread_start_us":125,"threads_started":1}
I20260926 16:46:14.037989 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:14.030827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41036 (local address 127.21.12.65:33915)
0926 16:46:14.031334 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:14.031349 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:14.031371 (+ 22us) server_negotiation.cc:407] Connection header received
0926 16:46:14.031433 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:14.031437 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:14.031586 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:14.031746 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:46:14.032212 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:14.033403 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:14.036485 (+ 3082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:14.037351 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:14.037471 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:14.037617 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:14.037727 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 16:46:14.037835 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":103,"threads_started":1}
I20260926 16:46:14.038002 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:14.030437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:33915 (local address 127.0.0.1:41036)
0926 16:46:14.031269 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:14.031281 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:14.031349 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:14.031907 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:14.031910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:14.031941 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:46:14.032082 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:14.032090 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:14.033570 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:14.033573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:14.034941 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:14.034949 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:14.037486 (+ 2537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:14.037490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:14.037588 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:14.037611 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 16:46:14.037688 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 16:46:14.037795 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"mutex_wait_us":157,"spinlock_wait_cycles":5760,"thread_start_us":88,"threads_started":1}
I20260926 16:46:14.038859 7671 tablet_service.cc:1467] Processing CreateTablet for tablet 4a108cd25ef546ef86b6617ef3cf53b7 (DEFAULT_TABLE table=test-table [id=ec8a4bf3d541485cae031c90b7da1af6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:46:14.039783 7671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a108cd25ef546ef86b6617ef3cf53b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:14.043316 7535 tablet_service.cc:1467] Processing CreateTablet for tablet 4a108cd25ef546ef86b6617ef3cf53b7 (DEFAULT_TABLE table=test-table [id=ec8a4bf3d541485cae031c90b7da1af6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:46:14.043319 7400 tablet_service.cc:1467] Processing CreateTablet for tablet 4a108cd25ef546ef86b6617ef3cf53b7 (DEFAULT_TABLE table=test-table [id=ec8a4bf3d541485cae031c90b7da1af6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:46:14.044333 7400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a108cd25ef546ef86b6617ef3cf53b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:14.044333 7535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a108cd25ef546ef86b6617ef3cf53b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:14.050101 7755 tablet_bootstrap.cc:492] T 4a108cd25ef546ef86b6617ef3cf53b7 P 6201cadefff4480ea94cf0537bb3f328: Bootstrap starting.
I20260926 16:46:14.051839 7757 tablet_bootstrap.cc:492] T 4a108cd25ef546ef86b6617ef3cf53b7 P 05f6c54cd0eb4dbca98007b1dbd409dd: Bootstrap starting.
I20260926 16:46:14.052258 7755 tablet_bootstrap.cc:654] T 4a108cd25ef546ef86b6617ef3cf53b7 P 6201cadefff4480ea94cf0537bb3f328: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:14.052232 7756 tablet_bootstrap.cc:492] T 4a108cd25ef546ef86b6617ef3cf53b7 P 7df6d6270bf041e59390794d64e9ccea: Bootstrap starting.
I20260926 16:46:14.053608 7755 log.cc:824] T 4a108cd25ef546ef86b6617ef3cf53b7 P 6201cadefff4480ea94cf0537bb3f328: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:14.054423 7757 tablet_bootstrap.cc:654] T 4a108cd25ef546ef86b6617ef3cf53b7 P 05f6c54cd0eb4dbca98007b1dbd409dd: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:14.055732 7757 log.cc:824] T 4a108cd25ef546ef86b6617ef3cf53b7 P 05f6c54cd0eb4dbca98007b1dbd409dd: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:14.055898 7755 tablet_bootstrap.cc:492] T 4a108cd25ef546ef86b6617ef3cf53b7 P 6201cadefff4480ea94cf0537bb3f328: No bootstrap required, opened a new log
I20260926 16:46:14.056028 7756 tablet_bootstrap.cc:654] T 4a108cd25ef546ef86b6617ef3cf53b7 P 7df6d6270bf041e59390794d64e9ccea: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:14.056106 7755 ts_tablet_manager.cc:1397] T 4a108cd25ef546ef86b6617ef3cf53b7 P 6201cadefff4480ea94cf0537bb3f328: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260926 16:46:14.056890 7756 log.cc:824] T 4a108cd25ef546ef86b6617ef3cf53b7 P 7df6d6270bf041e59390794d64e9ccea: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:14.057750 7757 tablet_bootstrap.cc:492] T 4a108cd25ef546ef86b6617ef3cf53b7 P 05f6c54cd0eb4dbca98007b1dbd409dd: No bootstrap required, opened a new log
I20260926 16:46:14.057958 7757 ts_tablet_manager.cc:1397] T 4a108cd25ef546ef86b6617ef3cf53b7 P 05f6c54cd0eb4dbca98007b1dbd409dd: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260926 16:46:14.058858 7756 tablet_bootstrap.cc:492] T 4a108cd25ef546ef86b6617ef3cf53b7 P 7df6d6270bf041e59390794d64e9ccea: No bootstrap required, opened a new log
I20260926 16:46:14.059039 7756 ts_tablet_manager.cc:1397] T 4a108cd25ef546ef86b6617ef3cf53b7 P 7df6d6270bf041e59390794d64e9ccea: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260926 16:46:14.059726 7755 raft_consensus.cc:348] T 4a108cd25ef546ef86b6617ef3cf53b7 P 6201cadefff4480ea94cf0537bb3f328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6201cadefff4480ea94cf0537bb3f328" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42741 } } peers { permanent_uuid: "7df6d6270bf041e59390794d64e9ccea" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37803 } } peers { permanent_uuid: "05f6c54cd0eb4dbca98007b1dbd409dd" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 33915 } }
I20260926 16:46:14.060024 7755 raft_consensus.cc:374] T 4a108cd25ef546ef86b6617ef3cf53b7 P 6201cadefff4480ea94cf0537bb3f328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:46:14.060092 7755 raft_consensus.cc:729] T 4a108cd25ef546ef86b6617ef3cf53b7 P 6201cadefff4480ea94cf0537bb3f328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6201cadefff4480ea94cf0537bb3f328, State: Initialized, Role: FOLLOWER
I20260926 16:46:14.060626 7755 consensus_queue.cc:260] T 4a108cd25ef546ef86b6617ef3cf53b7 P 6201cadefff4480ea94cf0537bb3f328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6201cadefff4480ea94cf0537bb3f328" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42741 } } peers { permanent_uuid: "7df6d6270bf041e59390794d64e9ccea" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37803 } } peers { permanent_uuid: "05f6c54cd0eb4dbca98007b1dbd409dd" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 33915 } }
I20260926 16:46:14.061264 7738 heartbeater.cc:503] Master 127.21.12.126:43481 was elected leader, sending a full tablet report...
I20260926 16:46:14.061935 7757 raft_consensus.cc:348] T 4a108cd25ef546ef86b6617ef3cf53b7 P 05f6c54cd0eb4dbca98007b1dbd409dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6201cadefff4480ea94cf0537bb3f328" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42741 } } peers { permanent_uuid: "7df6d6270bf041e59390794d64e9ccea" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37803 } } peers { permanent_uuid: "05f6c54cd0eb4dbca98007b1dbd409dd" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 33915 } }
I20260926 16:46:14.062242 7757 raft_consensus.cc:374] T 4a108cd25ef546ef86b6617ef3cf53b7 P 05f6c54cd0eb4dbca98007b1dbd409dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:46:14.062294 7757 raft_consensus.cc:729] T 4a108cd25ef546ef86b6617ef3cf53b7 P 05f6c54cd0eb4dbca98007b1dbd409dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f6c54cd0eb4dbca98007b1dbd409dd, State: Initialized, Role: FOLLOWER
I20260926 16:46:14.062657 7755 ts_tablet_manager.cc:1428] T 4a108cd25ef546ef86b6617ef3cf53b7 P 6201cadefff4480ea94cf0537bb3f328: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 16:46:14.062731 7757 consensus_queue.cc:260] T 4a108cd25ef546ef86b6617ef3cf53b7 P 05f6c54cd0eb4dbca98007b1dbd409dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6201cadefff4480ea94cf0537bb3f328" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42741 } } peers { permanent_uuid: "7df6d6270bf041e59390794d64e9ccea" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37803 } } peers { permanent_uuid: "05f6c54cd0eb4dbca98007b1dbd409dd" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 33915 } }
I20260926 16:46:14.062889 7756 raft_consensus.cc:348] T 4a108cd25ef546ef86b6617ef3cf53b7 P 7df6d6270bf041e59390794d64e9ccea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6201cadefff4480ea94cf0537bb3f328" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42741 } } peers { permanent_uuid: "7df6d6270bf041e59390794d64e9ccea" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37803 } } peers { permanent_uuid: "05f6c54cd0eb4dbca98007b1dbd409dd" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 33915 } }
I20260926 16:46:14.063135 7756 raft_consensus.cc:374] T 4a108cd25ef546ef86b6617ef3cf53b7 P 7df6d6270bf041e59390794d64e9ccea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:46:14.063181 7756 raft_consensus.cc:729] T 4a108cd25ef546ef86b6617ef3cf53b7 P 7df6d6270bf041e59390794d64e9ccea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df6d6270bf041e59390794d64e9ccea, State: Initialized, Role: FOLLOWER
I20260926 16:46:14.063405 7467 heartbeater.cc:503] Master 127.21.12.126:43481 was elected leader, sending a full tablet report...
I20260926 16:46:14.063596 7756 consensus_queue.cc:260] T 4a108cd25ef546ef86b6617ef3cf53b7 P 7df6d6270bf041e59390794d64e9ccea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6201cadefff4480ea94cf0537bb3f328" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42741 } } peers { permanent_uuid: "7df6d6270bf041e59390794d64e9ccea" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37803 } } peers { permanent_uuid: "05f6c54cd0eb4dbca98007b1dbd409dd" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 33915 } }
I20260926 16:46:14.063738 7757 ts_tablet_manager.cc:1428] T 4a108cd25ef546ef86b6617ef3cf53b7 P 05f6c54cd0eb4dbca98007b1dbd409dd: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260926 16:46:14.064644 7756 ts_tablet_manager.cc:1428] T 4a108cd25ef546ef86b6617ef3cf53b7 P 7df6d6270bf041e59390794d64e9ccea: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 16:46:14.064978 7601 heartbeater.cc:503] Master 127.21.12.126:43481 was elected leader, sending a full tablet report...
I20260926 16:46:14.131024 7762 raft_consensus.cc:482] T 4a108cd25ef546ef86b6617ef3cf53b7 P 05f6c54cd0eb4dbca98007b1dbd409dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:46:14.131274 7762 raft_consensus.cc:504] T 4a108cd25ef546ef86b6617ef3cf53b7 P 05f6c54cd0eb4dbca98007b1dbd409dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6201cadefff4480ea94cf0537bb3f328" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42741 } } peers { permanent_uuid: "7df6d6270bf041e59390794d64e9ccea" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37803 } } peers { permanent_uuid: "05f6c54cd0eb4dbca98007b1dbd409dd" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 33915 } }
I20260926 16:46:14.131812 7763 raft_consensus.cc:482] T 4a108cd25ef546ef86b6617ef3cf53b7 P 7df6d6270bf041e59390794d64e9ccea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:46:14.132059 7763 raft_consensus.cc:504] T 4a108cd25ef546ef86b6617ef3cf53b7 P 7df6d6270bf041e59390794d64e9ccea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6201cadefff4480ea94cf0537bb3f328" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42741 } } peers { permanent_uuid: "7df6d6270bf041e59390794d64e9ccea" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37803 } } peers { permanent_uuid: "05f6c54cd0eb4dbca98007b1dbd409dd" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 33915 } }
I20260926 16:46:14.133090 7762 leader_election.cc:288] T 4a108cd25ef546ef86b6617ef3cf53b7 P 05f6c54cd0eb4dbca98007b1dbd409dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6201cadefff4480ea94cf0537bb3f328 (127.21.12.67:42741), 7df6d6270bf041e59390794d64e9ccea (127.21.12.66:37803)
I20260926 16:46:14.133368 7763 leader_election.cc:288] T 4a108cd25ef546ef86b6617ef3cf53b7 P 7df6d6270bf041e59390794d64e9ccea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6201cadefff4480ea94cf0537bb3f328 (127.21.12.67:42741), 05f6c54cd0eb4dbca98007b1dbd409dd (127.21.12.65:33915)
I20260926 16:46:14.137660 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:14.132903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:43087 (local address 127.21.12.66:37803)
0926 16:46:14.133036 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:14.133040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:14.133318 (+ 278us) server_negotiation.cc:407] Connection header received
0926 16:46:14.133627 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:14.133630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:14.133687 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:14.133735 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:46:14.134219 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:14.134952 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:14.136386 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:14.136962 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:14.137070 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:14.137135 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:14.137457 (+ 322us) server_negotiation.cc:299] Negotiation successful
0926 16:46:14.137528 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":56}
I20260926 16:46:14.138123 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:14.132825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:37803 (local address 127.21.12.65:43087)
0926 16:46:14.133230 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:14.133248 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:14.133401 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:14.133912 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:14.133915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:14.133945 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:46:14.134079 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:14.134090 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:14.135081 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:14.135084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:14.136273 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:14.136281 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:14.137197 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:14.137201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:14.137301 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:14.137324 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 16:46:14.137856 (+ 532us) client_negotiation.cc:240] Negotiation successful
0926 16:46:14.137974 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":28,"thread_start_us":107,"threads_started":1}
I20260926 16:46:14.138267 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:14.133046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:45721 (local address 127.21.12.67:42741)
0926 16:46:14.133183 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:14.133188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:14.133788 (+ 600us) server_negotiation.cc:407] Connection header received
0926 16:46:14.134029 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:14.134032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:14.134091 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:14.134149 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:46:14.134626 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:14.135360 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:14.136850 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:14.137498 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:14.137619 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:14.137671 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:14.138054 (+ 383us) server_negotiation.cc:299] Negotiation successful
0926 16:46:14.138132 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":58}
I20260926 16:46:14.138645 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:14.132825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.67:42741 (local address 127.21.12.65:45721)
0926 16:46:14.133666 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:14.133683 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:14.133876 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:14.134305 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:14.134309 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:14.134328 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:46:14.134486 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:14.134497 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:14.135502 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:14.135506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:14.136696 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:14.136708 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:14.137733 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:14.137737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:14.137871 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:14.137934 (+ 63us) client_negotiation.cc:769] Sending connection context
0926 16:46:14.138415 (+ 481us) client_negotiation.cc:240] Negotiation successful
0926 16:46:14.138488 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"mutex_wait_us":378,"thread_start_us":123,"threads_started":1}
I20260926 16:46:14.138691 7555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a108cd25ef546ef86b6617ef3cf53b7" candidate_uuid: "05f6c54cd0eb4dbca98007b1dbd409dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df6d6270bf041e59390794d64e9ccea" is_pre_election: true
I20260926 16:46:14.138862 7603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:14.133480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.67:42741 (local address 127.21.12.66:60997)
0926 16:46:14.133632 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:14.133644 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:14.133694 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:14.134562 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:14.134565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:14.134583 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:46:14.134676 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:14.134681 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:14.135785 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:14.135788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:14.137070 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:14.137078 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:14.138522 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:14.138526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:14.138618 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:14.138640 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 16:46:14.138689 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 16:46:14.138754 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":60}
I20260926 16:46:14.138893 7555 raft_consensus.cc:2454] T 4a108cd25ef546ef86b6617ef3cf53b7 P 7df6d6270bf041e59390794d64e9ccea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f6c54cd0eb4dbca98007b1dbd409dd in term 0.
I20260926 16:46:14.139266 7691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a108cd25ef546ef86b6617ef3cf53b7" candidate_uuid: "05f6c54cd0eb4dbca98007b1dbd409dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6201cadefff4480ea94cf0537bb3f328" is_pre_election: true
I20260926 16:46:14.139356 7355 leader_election.cc:302] T 4a108cd25ef546ef86b6617ef3cf53b7 P 05f6c54cd0eb4dbca98007b1dbd409dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f6c54cd0eb4dbca98007b1dbd409dd, 7df6d6270bf041e59390794d64e9ccea; no voters:
I20260926 16:46:14.139590 7691 raft_consensus.cc:2454] T 4a108cd25ef546ef86b6617ef3cf53b7 P 6201cadefff4480ea94cf0537bb3f328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f6c54cd0eb4dbca98007b1dbd409dd in term 0.
I20260926 16:46:14.139707 7762 raft_consensus.cc:2781] T 4a108cd25ef546ef86b6617ef3cf53b7 P 05f6c54cd0eb4dbca98007b1dbd409dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:46:14.139801 7762 raft_consensus.cc:482] T 4a108cd25ef546ef86b6617ef3cf53b7 P 05f6c54cd0eb4dbca98007b1dbd409dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:46:14.139856 7762 raft_consensus.cc:3037] T 4a108cd25ef546ef86b6617ef3cf53b7 P 05f6c54cd0eb4dbca98007b1dbd409dd [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:14.139946 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:14.134129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:36323 (local address 127.21.12.65:33915)
0926 16:46:14.134232 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:14.134239 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:14.134500 (+ 261us) server_negotiation.cc:407] Connection header received
0926 16:46:14.134685 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:14.134688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:14.134729 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:14.134777 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:46:14.135189 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:14.135900 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:14.137998 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:14.138532 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:14.138621 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:14.138682 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:14.139736 (+ 1054us) server_negotiation.cc:299] Negotiation successful
0926 16:46:14.139812 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 16:46:14.140268 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:14.133724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:33915 (local address 127.21.12.66:36323)
0926 16:46:14.134414 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:14.134430 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:14.134570 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:14.134920 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:14.134923 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:14.134941 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:46:14.135062 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:14.135068 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:14.136049 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:14.136052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:14.137854 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:14.137865 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:14.138908 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:14.138913 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:14.139524 (+ 611us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:14.139562 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 16:46:14.139701 (+ 139us) client_negotiation.cc:240] Negotiation successful
0926 16:46:14.140114 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"mutex_wait_us":4,"thread_start_us":130,"threads_started":1}
I20260926 16:46:14.139622 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:14.133943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:60997 (local address 127.21.12.67:42741)
0926 16:46:14.134333 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:14.134342 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:14.134354 (+ 12us) server_negotiation.cc:407] Connection header received
0926 16:46:14.134392 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:14.134394 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:14.134445 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:14.134765 (+ 320us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:46:14.134845 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:14.135655 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:14.137187 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:14.138104 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:14.139334 (+ 1230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:14.139390 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:14.139424 (+ 34us) server_negotiation.cc:299] Negotiation successful
0926 16:46:14.139488 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":326,"thread_start_us":119,"threads_started":1}
I20260926 16:46:14.140887 7691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a108cd25ef546ef86b6617ef3cf53b7" candidate_uuid: "7df6d6270bf041e59390794d64e9ccea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6201cadefff4480ea94cf0537bb3f328" is_pre_election: true
I20260926 16:46:14.140933 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a108cd25ef546ef86b6617ef3cf53b7" candidate_uuid: "7df6d6270bf041e59390794d64e9ccea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f6c54cd0eb4dbca98007b1dbd409dd" is_pre_election: true
I20260926 16:46:14.141040 7691 raft_consensus.cc:2454] T 4a108cd25ef546ef86b6617ef3cf53b7 P 6201cadefff4480ea94cf0537bb3f328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7df6d6270bf041e59390794d64e9ccea in term 0.
I20260926 16:46:14.141254 7762 raft_consensus.cc:504] T 4a108cd25ef546ef86b6617ef3cf53b7 P 05f6c54cd0eb4dbca98007b1dbd409dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6201cadefff4480ea94cf0537bb3f328" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42741 } } peers { permanent_uuid: "7df6d6270bf041e59390794d64e9ccea" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37803 } } peers { permanent_uuid: "05f6c54cd0eb4dbca98007b1dbd409dd" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 33915 } }
I20260926 16:46:14.141498 7420 raft_consensus.cc:2379] T 4a108cd25ef546ef86b6617ef3cf53b7 P 05f6c54cd0eb4dbca98007b1dbd409dd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7df6d6270bf041e59390794d64e9ccea in current term 1: Already voted for candidate 05f6c54cd0eb4dbca98007b1dbd409dd in this term.
I20260926 16:46:14.141537 7490 leader_election.cc:302] T 4a108cd25ef546ef86b6617ef3cf53b7 P 7df6d6270bf041e59390794d64e9ccea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6201cadefff4480ea94cf0537bb3f328, 7df6d6270bf041e59390794d64e9ccea; no voters:
I20260926 16:46:14.141620 7762 leader_election.cc:288] T 4a108cd25ef546ef86b6617ef3cf53b7 P 05f6c54cd0eb4dbca98007b1dbd409dd [CANDIDATE]: Term 1 election: Requested vote from peers 6201cadefff4480ea94cf0537bb3f328 (127.21.12.67:42741), 7df6d6270bf041e59390794d64e9ccea (127.21.12.66:37803)
I20260926 16:46:14.141865 7763 raft_consensus.cc:2781] T 4a108cd25ef546ef86b6617ef3cf53b7 P 7df6d6270bf041e59390794d64e9ccea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:46:14.142022 7763 raft_consensus.cc:482] T 4a108cd25ef546ef86b6617ef3cf53b7 P 7df6d6270bf041e59390794d64e9ccea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:46:14.142079 7763 raft_consensus.cc:3037] T 4a108cd25ef546ef86b6617ef3cf53b7 P 7df6d6270bf041e59390794d64e9ccea [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:14.142042 7555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a108cd25ef546ef86b6617ef3cf53b7" candidate_uuid: "05f6c54cd0eb4dbca98007b1dbd409dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df6d6270bf041e59390794d64e9ccea"
I20260926 16:46:14.142285 7691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a108cd25ef546ef86b6617ef3cf53b7" candidate_uuid: "05f6c54cd0eb4dbca98007b1dbd409dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6201cadefff4480ea94cf0537bb3f328"
I20260926 16:46:14.142398 7691 raft_consensus.cc:3037] T 4a108cd25ef546ef86b6617ef3cf53b7 P 6201cadefff4480ea94cf0537bb3f328 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:14.143338 7763 raft_consensus.cc:504] T 4a108cd25ef546ef86b6617ef3cf53b7 P 7df6d6270bf041e59390794d64e9ccea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6201cadefff4480ea94cf0537bb3f328" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42741 } } peers { permanent_uuid: "7df6d6270bf041e59390794d64e9ccea" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37803 } } peers { permanent_uuid: "05f6c54cd0eb4dbca98007b1dbd409dd" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 33915 } }
I20260926 16:46:14.143627 7555 raft_consensus.cc:2379] T 4a108cd25ef546ef86b6617ef3cf53b7 P 7df6d6270bf041e59390794d64e9ccea [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 05f6c54cd0eb4dbca98007b1dbd409dd in current term 1: Already voted for candidate 7df6d6270bf041e59390794d64e9ccea in this term.
I20260926 16:46:14.143680 7691 raft_consensus.cc:2454] T 4a108cd25ef546ef86b6617ef3cf53b7 P 6201cadefff4480ea94cf0537bb3f328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f6c54cd0eb4dbca98007b1dbd409dd in term 1.
I20260926 16:46:14.143725 7763 leader_election.cc:288] T 4a108cd25ef546ef86b6617ef3cf53b7 P 7df6d6270bf041e59390794d64e9ccea [CANDIDATE]: Term 1 election: Requested vote from peers 6201cadefff4480ea94cf0537bb3f328 (127.21.12.67:42741), 05f6c54cd0eb4dbca98007b1dbd409dd (127.21.12.65:33915)
I20260926 16:46:14.144011 7354 leader_election.cc:302] T 4a108cd25ef546ef86b6617ef3cf53b7 P 05f6c54cd0eb4dbca98007b1dbd409dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f6c54cd0eb4dbca98007b1dbd409dd, 6201cadefff4480ea94cf0537bb3f328; no voters:
I20260926 16:46:14.144081 7691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a108cd25ef546ef86b6617ef3cf53b7" candidate_uuid: "7df6d6270bf041e59390794d64e9ccea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6201cadefff4480ea94cf0537bb3f328"
I20260926 16:46:14.144173 7762 raft_consensus.cc:2781] T 4a108cd25ef546ef86b6617ef3cf53b7 P 05f6c54cd0eb4dbca98007b1dbd409dd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:46:14.144250 7691 raft_consensus.cc:2379] T 4a108cd25ef546ef86b6617ef3cf53b7 P 6201cadefff4480ea94cf0537bb3f328 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7df6d6270bf041e59390794d64e9ccea in current term 1: Already voted for candidate 05f6c54cd0eb4dbca98007b1dbd409dd in this term.
I20260926 16:46:14.144378 7762 raft_consensus.cc:686] T 4a108cd25ef546ef86b6617ef3cf53b7 P 05f6c54cd0eb4dbca98007b1dbd409dd [term 1 LEADER]: Becoming Leader. State: Replica: 05f6c54cd0eb4dbca98007b1dbd409dd, State: Running, Role: LEADER
I20260926 16:46:14.144672 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a108cd25ef546ef86b6617ef3cf53b7" candidate_uuid: "7df6d6270bf041e59390794d64e9ccea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f6c54cd0eb4dbca98007b1dbd409dd"
I20260926 16:46:14.144709 7762 consensus_queue.cc:237] T 4a108cd25ef546ef86b6617ef3cf53b7 P 05f6c54cd0eb4dbca98007b1dbd409dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6201cadefff4480ea94cf0537bb3f328" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42741 } } peers { permanent_uuid: "7df6d6270bf041e59390794d64e9ccea" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37803 } } peers { permanent_uuid: "05f6c54cd0eb4dbca98007b1dbd409dd" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 33915 } }
I20260926 16:46:14.144991 7491 leader_election.cc:302] T 4a108cd25ef546ef86b6617ef3cf53b7 P 7df6d6270bf041e59390794d64e9ccea [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7df6d6270bf041e59390794d64e9ccea; no voters: 05f6c54cd0eb4dbca98007b1dbd409dd, 6201cadefff4480ea94cf0537bb3f328
I20260926 16:46:14.145185 7763 raft_consensus.cc:2726] T 4a108cd25ef546ef86b6617ef3cf53b7 P 7df6d6270bf041e59390794d64e9ccea [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260926 16:46:14.147861 7274 catalog_manager.cc:5495] T 4a108cd25ef546ef86b6617ef3cf53b7 P 05f6c54cd0eb4dbca98007b1dbd409dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 05f6c54cd0eb4dbca98007b1dbd409dd (127.21.12.65). New cstate: current_term: 1 leader_uuid: "05f6c54cd0eb4dbca98007b1dbd409dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6201cadefff4480ea94cf0537bb3f328" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7df6d6270bf041e59390794d64e9ccea" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f6c54cd0eb4dbca98007b1dbd409dd" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 33915 } health_report { overall_health: HEALTHY } } }
I20260926 16:46:14.158449 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:14.155118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41042 (local address 127.21.12.65:33915)
0926 16:46:14.155234 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:14.155237 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:14.155284 (+ 47us) server_negotiation.cc:407] Connection header received
0926 16:46:14.155440 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:14.155442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:14.155493 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:14.155579 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:46:14.155797 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:14.156284 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:14.157109 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:14.157306 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:14.157379 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:14.157696 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:46:14.158127 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:46:14.158239 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:14.158323 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 16:46:14.158351 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 16:46:14.163913 7555 raft_consensus.cc:1261] T 4a108cd25ef546ef86b6617ef3cf53b7 P 7df6d6270bf041e59390794d64e9ccea [term 1 FOLLOWER]: Refusing update from remote peer 05f6c54cd0eb4dbca98007b1dbd409dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:46:14.163939 7691 raft_consensus.cc:1261] T 4a108cd25ef546ef86b6617ef3cf53b7 P 6201cadefff4480ea94cf0537bb3f328 [term 1 FOLLOWER]: Refusing update from remote peer 05f6c54cd0eb4dbca98007b1dbd409dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:46:14.164559 7762 consensus_queue.cc:1035] T 4a108cd25ef546ef86b6617ef3cf53b7 P 05f6c54cd0eb4dbca98007b1dbd409dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6201cadefff4480ea94cf0537bb3f328" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:46:14.164794 7768 consensus_queue.cc:1035] T 4a108cd25ef546ef86b6617ef3cf53b7 P 05f6c54cd0eb4dbca98007b1dbd409dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "7df6d6270bf041e59390794d64e9ccea" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 37803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:46:14.171885 7771 mvcc.cc:205] Tried to move back new op lower bound from 7333647049368956928 to 7333647049299689472. Current Snapshot: MvccSnapshot[applied={T|T < 7333647049368956928}]
I20260926 16:46:14.174623 7772 mvcc.cc:205] Tried to move back new op lower bound from 7333647049368956928 to 7333647049299689472. Current Snapshot: MvccSnapshot[applied={T|T < 7333647049368956928}]
I20260926 16:46:14.180780 7271 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:46:14.180944 7271 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:46:14.183226 7271 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ccdb698b9b9648cd853fcc6ff43f626c: Sending DeleteTablet for 3 replicas of tablet 4a108cd25ef546ef86b6617ef3cf53b7
I20260926 16:46:14.183904 7671 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a108cd25ef546ef86b6617ef3cf53b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:46:14 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:46368
I20260926 16:46:14.183918 7535 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a108cd25ef546ef86b6617ef3cf53b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:46:14 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:44070
I20260926 16:46:14.184263 7400 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a108cd25ef546ef86b6617ef3cf53b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:46:14 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:41036
I20260926 16:46:14.184405 7780 tablet_replica.cc:330] T 4a108cd25ef546ef86b6617ef3cf53b7 P 7df6d6270bf041e59390794d64e9ccea: stopping tablet replica
I20260926 16:46:14.184682 7782 tablet_replica.cc:330] T 4a108cd25ef546ef86b6617ef3cf53b7 P 05f6c54cd0eb4dbca98007b1dbd409dd: stopping tablet replica
I20260926 16:46:14.184705 7780 raft_consensus.cc:2229] T 4a108cd25ef546ef86b6617ef3cf53b7 P 7df6d6270bf041e59390794d64e9ccea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:46:14.184983 7781 tablet_replica.cc:330] T 4a108cd25ef546ef86b6617ef3cf53b7 P 6201cadefff4480ea94cf0537bb3f328: stopping tablet replica
I20260926 16:46:14.185004 7780 raft_consensus.cc:2258] T 4a108cd25ef546ef86b6617ef3cf53b7 P 7df6d6270bf041e59390794d64e9ccea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:46:14.185035 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 7336
I20260926 16:46:14.185364 7781 raft_consensus.cc:2229] T 4a108cd25ef546ef86b6617ef3cf53b7 P 6201cadefff4480ea94cf0537bb3f328 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:46:14.185716 7781 raft_consensus.cc:2258] T 4a108cd25ef546ef86b6617ef3cf53b7 P 6201cadefff4480ea94cf0537bb3f328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:46:14.186527 7780 ts_tablet_manager.cc:1905] T 4a108cd25ef546ef86b6617ef3cf53b7 P 7df6d6270bf041e59390794d64e9ccea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:46:14.187933 7781 ts_tablet_manager.cc:1905] T 4a108cd25ef546ef86b6617ef3cf53b7 P 6201cadefff4480ea94cf0537bb3f328: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:46:14.189276 7780 ts_tablet_manager.cc:1918] T 4a108cd25ef546ef86b6617ef3cf53b7 P 7df6d6270bf041e59390794d64e9ccea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:46:14.189383 7780 log.cc:1196] T 4a108cd25ef546ef86b6617ef3cf53b7 P 7df6d6270bf041e59390794d64e9ccea: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-1/wal/wals/4a108cd25ef546ef86b6617ef3cf53b7
I20260926 16:46:14.189774 7780 ts_tablet_manager.cc:1939] T 4a108cd25ef546ef86b6617ef3cf53b7 P 7df6d6270bf041e59390794d64e9ccea: Deleting consensus metadata
I20260926 16:46:14.190609 7781 ts_tablet_manager.cc:1918] T 4a108cd25ef546ef86b6617ef3cf53b7 P 6201cadefff4480ea94cf0537bb3f328: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:46:14.190701 7781 log.cc:1196] T 4a108cd25ef546ef86b6617ef3cf53b7 P 6201cadefff4480ea94cf0537bb3f328: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440924937351-21553-0/minicluster-data/ts-2/wal/wals/4a108cd25ef546ef86b6617ef3cf53b7
I20260926 16:46:14.191038 7261 catalog_manager.cc:4841] TS 7df6d6270bf041e59390794d64e9ccea (127.21.12.66:37803): tablet 4a108cd25ef546ef86b6617ef3cf53b7 (table test-table [id=ec8a4bf3d541485cae031c90b7da1af6]) successfully deleted
I20260926 16:46:14.191365 7781 ts_tablet_manager.cc:1939] T 4a108cd25ef546ef86b6617ef3cf53b7 P 6201cadefff4480ea94cf0537bb3f328: Deleting consensus metadata
I20260926 16:46:14.192548 7260 catalog_manager.cc:4841] TS 6201cadefff4480ea94cf0537bb3f328 (127.21.12.67:42741): tablet 4a108cd25ef546ef86b6617ef3cf53b7 (table test-table [id=ec8a4bf3d541485cae031c90b7da1af6]) successfully deleted
W20260926 16:46:14.192802 7261 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.12.65:33915 (error 108)
I20260926 16:46:14.192812 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 7472
I20260926 16:46:14.193876 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:14.193555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:33915 (local address 127.0.0.1:41050)
0926 16:46:14.193677 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:14.193743 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.65:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 16:46:14.194110 7261 catalog_manager.cc:4584] TS 05f6c54cd0eb4dbca98007b1dbd409dd (127.21.12.65:33915): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4a108cd25ef546ef86b6617ef3cf53b7: Network error: Client connection negotiation failed: client connection to 127.21.12.65:33915: connect: Connection refused (error 111)
I20260926 16:46:14.198895 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 7607
I20260926 16:46:14.204267 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 7241
2026-09-26T16:46:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2263 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T16:46:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:46:14Z Disabled control of system clock
I20260926 16:46:14.234899 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:38209
--webserver_interface=127.21.12.126
--webserver_port=0
--builtin_ntp_servers=127.21.12.84:40647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:38209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 16:46:14.344445 7787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:14.344700 7787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:14.344735 7787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:14.349205 7787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:46:14.349272 7787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:14.349288 7787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:14.349305 7787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:46:14.349320 7787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:46:14.354151 7787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:40647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:38209
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:38209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:46:14.355194 7787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:14.356547 7787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:14.364979 7795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:14.364979 7793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:14.365048 7792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:14.365096 7787 server_base.cc:1034] running on GCE node
I20260926 16:46:14.365836 7787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:14.366976 7787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:14.368167 7787 hybrid_clock.cc:648] HybridClock initialized: now 1790441174368151 us; error 28 us; skew 500 ppm
I20260926 16:46:14.370095 7787 webserver.cc:460] Webserver started at http://127.21.12.126:43721/ using document root <none> and password file <none>
I20260926 16:46:14.370666 7787 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:14.370754 7787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:14.370973 7787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:46:14.372977 7787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "40d76984fa934938a0bd558587c45bea"
format_stamp: "Formatted at 2026-09-26 16:46:14 on dist-test-slave-n1hw"
server_key: "f0a92407fd54c1bd8306d1a0f1ef462d"
server_key_iv: "275650e61aa0e4d1573ec3b9810bc248"
server_key_version: "encryptionkey@0"
I20260926 16:46:14.373545 7787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "40d76984fa934938a0bd558587c45bea"
format_stamp: "Formatted at 2026-09-26 16:46:14 on dist-test-slave-n1hw"
server_key: "f0a92407fd54c1bd8306d1a0f1ef462d"
server_key_iv: "275650e61aa0e4d1573ec3b9810bc248"
server_key_version: "encryptionkey@0"
I20260926 16:46:14.377112 7787 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 16:46:14.379484 7801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:14.380683 7787 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260926 16:46:14.380813 7787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "40d76984fa934938a0bd558587c45bea"
format_stamp: "Formatted at 2026-09-26 16:46:14 on dist-test-slave-n1hw"
server_key: "f0a92407fd54c1bd8306d1a0f1ef462d"
server_key_iv: "275650e61aa0e4d1573ec3b9810bc248"
server_key_version: "encryptionkey@0"
I20260926 16:46:14.380918 7787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:14.404048 7787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:14.404742 7787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:14.404932 7787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:14.411414 7787 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.126:38209
I20260926 16:46:14.411437 7854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.126:38209 every 8 connection(s)
I20260926 16:46:14.412331 7787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:46:14.415560 7855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:14.420498 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 7787
I20260926 16:46:14.420645 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:46:14.420989 21553 external_mini_cluster.cc:1428] Setting key da830e2dd77eeb97a92cfb8adbc56c07
I20260926 16:46:14.421526 7855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40d76984fa934938a0bd558587c45bea: Bootstrap starting.
I20260926 16:46:14.424113 7855 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40d76984fa934938a0bd558587c45bea: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:14.424894 7855 log.cc:824] T 00000000000000000000000000000000 P 40d76984fa934938a0bd558587c45bea: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:14.426891 7855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40d76984fa934938a0bd558587c45bea: No bootstrap required, opened a new log
I20260926 16:46:14.430663 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:14.422535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57646 (local address 127.21.12.126:38209)
0926 16:46:14.423238 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:14.423253 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:14.423287 (+ 34us) server_negotiation.cc:407] Connection header received
0926 16:46:14.423911 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:14.423928 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:14.424481 (+ 553us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:14.424858 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:14.425284 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:14.426378 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:14.427342 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:14.427823 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:14.427989 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:14.428842 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:14.428847 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:14.428861 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:46:14.429232 (+ 371us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:14.429287 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:46:14.429306 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:14.429421 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:14.429538 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 16:46:14.429840 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":165,"threads_started":1}
I20260926 16:46:14.430554 7855 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40d76984fa934938a0bd558587c45bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d76984fa934938a0bd558587c45bea" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 38209 } }
I20260926 16:46:14.430819 7855 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40d76984fa934938a0bd558587c45bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:46:14.430917 7855 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40d76984fa934938a0bd558587c45bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40d76984fa934938a0bd558587c45bea, State: Initialized, Role: FOLLOWER
I20260926 16:46:14.431416 7855 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40d76984fa934938a0bd558587c45bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d76984fa934938a0bd558587c45bea" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 38209 } }
I20260926 16:46:14.431576 7855 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40d76984fa934938a0bd558587c45bea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:46:14.431634 7855 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40d76984fa934938a0bd558587c45bea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:46:14.431725 7855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40d76984fa934938a0bd558587c45bea [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:14.432658 7855 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40d76984fa934938a0bd558587c45bea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d76984fa934938a0bd558587c45bea" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 38209 } }
I20260926 16:46:14.433053 7855 leader_election.cc:302] T 00000000000000000000000000000000 P 40d76984fa934938a0bd558587c45bea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40d76984fa934938a0bd558587c45bea; no voters:
I20260926 16:46:14.433400 7855 leader_election.cc:288] T 00000000000000000000000000000000 P 40d76984fa934938a0bd558587c45bea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:46:14.433609 7860 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40d76984fa934938a0bd558587c45bea [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:46:14.433854 7860 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40d76984fa934938a0bd558587c45bea [term 1 LEADER]: Becoming Leader. State: Replica: 40d76984fa934938a0bd558587c45bea, State: Running, Role: LEADER
I20260926 16:46:14.434186 7860 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40d76984fa934938a0bd558587c45bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d76984fa934938a0bd558587c45bea" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 38209 } }
I20260926 16:46:14.434777 7855 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40d76984fa934938a0bd558587c45bea [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:46:14.435979 7862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40d76984fa934938a0bd558587c45bea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40d76984fa934938a0bd558587c45bea. Latest consensus state: current_term: 1 leader_uuid: "40d76984fa934938a0bd558587c45bea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d76984fa934938a0bd558587c45bea" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 38209 } } }
I20260926 16:46:14.435979 7861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40d76984fa934938a0bd558587c45bea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40d76984fa934938a0bd558587c45bea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d76984fa934938a0bd558587c45bea" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 38209 } } }
I20260926 16:46:14.436165 7861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40d76984fa934938a0bd558587c45bea [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:14.436412 7862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40d76984fa934938a0bd558587c45bea [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:14.436581 7869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:46:14.439209 7869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:46:14.444664 7869 catalog_manager.cc:1347] Generated new cluster ID: 58a3e082cb2440028f7089757f5da7fb
I20260926 16:46:14.444729 7869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:46:14.456568 7869 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:46:14.457405 7869 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:46:14.464800 7869 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40d76984fa934938a0bd558587c45bea: Generated new TSK 0
I20260926 16:46:14.465456 7869 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:46:14.474785 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.65:0
--local_ip_for_outbound_sockets=127.21.12.65
--webserver_interface=127.21.12.65
--webserver_port=0
--tserver_master_addrs=127.21.12.126:38209
--builtin_ntp_servers=127.21.12.84:40647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 16:46:14.585716 7879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:14.586047 7879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:14.586122 7879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:14.590171 7879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:14.590237 7879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:14.590322 7879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.65
I20260926 16:46:14.595021 7879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:40647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.65
--webserver_port=0
--tserver_master_addrs=127.21.12.126:38209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.65
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:46:14.596104 7879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:14.597466 7879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:14.607617 7885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:14.607621 7884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:14.607683 7887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:14.607869 7879 server_base.cc:1034] running on GCE node
I20260926 16:46:14.608258 7879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:14.608909 7879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:14.610080 7879 hybrid_clock.cc:648] HybridClock initialized: now 1790441174610062 us; error 26 us; skew 500 ppm
I20260926 16:46:14.612025 7879 webserver.cc:460] Webserver started at http://127.21.12.65:35205/ using document root <none> and password file <none>
I20260926 16:46:14.612601 7879 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:14.612668 7879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:14.612926 7879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:46:14.615022 7879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-0/data/instance:
uuid: "9020895d8ce14cbb9dd3a67aedae0db0"
format_stamp: "Formatted at 2026-09-26 16:46:14 on dist-test-slave-n1hw"
server_key: "52ea0059e14eb128bada0ccfbb6818f8"
server_key_iv: "07c0d45ad33cb6c287625efb6cac6ad9"
server_key_version: "encryptionkey@0"
I20260926 16:46:14.615628 7879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance:
uuid: "9020895d8ce14cbb9dd3a67aedae0db0"
format_stamp: "Formatted at 2026-09-26 16:46:14 on dist-test-slave-n1hw"
server_key: "52ea0059e14eb128bada0ccfbb6818f8"
server_key_iv: "07c0d45ad33cb6c287625efb6cac6ad9"
server_key_version: "encryptionkey@0"
I20260926 16:46:14.619737 7879 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 16:46:14.622107 7893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:14.623406 7879 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:46:14.623546 7879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-0/wal
uuid: "9020895d8ce14cbb9dd3a67aedae0db0"
format_stamp: "Formatted at 2026-09-26 16:46:14 on dist-test-slave-n1hw"
server_key: "52ea0059e14eb128bada0ccfbb6818f8"
server_key_iv: "07c0d45ad33cb6c287625efb6cac6ad9"
server_key_version: "encryptionkey@0"
I20260926 16:46:14.623659 7879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:14.639237 7879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:14.639833 7879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:14.639985 7879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:14.640468 7879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:46:14.641388 7879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:46:14.641434 7879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:14.641472 7879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:46:14.641487 7879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:14.649511 7879 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.65:37805
I20260926 16:46:14.649581 8007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.65:37805 every 8 connection(s)
I20260926 16:46:14.650552 7879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
I20260926 16:46:14.651249 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 7879
I20260926 16:46:14.651374 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance
I20260926 16:46:14.651621 21553 external_mini_cluster.cc:1428] Setting key 78c02a73cb649b0290f026e5914232d2
I20260926 16:46:14.653672 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.66:0
--local_ip_for_outbound_sockets=127.21.12.66
--webserver_interface=127.21.12.66
--webserver_port=0
--tserver_master_addrs=127.21.12.126:38209
--builtin_ntp_servers=127.21.12.84:40647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 16:46:14.662331 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:14.652422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:36691 (local address 127.21.12.126:38209)
0926 16:46:14.652569 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:14.652574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:14.653867 (+ 1293us) server_negotiation.cc:407] Connection header received
0926 16:46:14.655851 (+ 1984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:14.655857 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:14.655923 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:14.656039 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:14.657171 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:14.657704 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:14.659649 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:14.659892 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:14.659986 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:14.661781 (+ 1795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:14.661789 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:14.661795 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:46:14.661814 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:14.661841 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:46:14.661850 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:14.661953 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:14.662160 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 16:46:14.662215 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 16:46:14.663594 8010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:14.652943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38209 (local address 127.21.12.65:36691)
0926 16:46:14.653684 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:14.653743 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:14.655627 (+ 1884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:14.656197 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:14.656207 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:14.656319 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:46:14.657002 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:14.657025 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:14.657898 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:14.657904 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:14.659505 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:14.659519 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:14.660073 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:14.660079 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:14.660334 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:14.661399 (+ 1065us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:46:14.661401 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:46:14.661461 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:46:14.661554 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:46:14.661969 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:46:14.661982 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:46:14.661990 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 16:46:14.662142 (+ 152us) client_negotiation.cc:240] Negotiation successful
0926 16:46:14.662469 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":107,"threads_started":1}
I20260926 16:46:14.664925 8008 heartbeater.cc:347] Connected to a master server at 127.21.12.126:38209
I20260926 16:46:14.665194 8008 heartbeater.cc:464] Registering TS with master...
I20260926 16:46:14.665874 8008 heartbeater.cc:511] Master 127.21.12.126:38209 requested a full tablet report, sending...
I20260926 16:46:14.667479 7818 ts_manager.cc:194] Registered new tserver with Master: 9020895d8ce14cbb9dd3a67aedae0db0 (127.21.12.65:37805)
I20260926 16:46:14.669001 7818 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.12.65:36691
W20260926 16:46:14.765395 8011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:14.765640 8011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:14.765676 8011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:14.769534 8011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:14.769586 8011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:14.769637 8011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.66
I20260926 16:46:14.774240 8011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:40647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.66
--webserver_port=0
--tserver_master_addrs=127.21.12.126:38209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.66
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:46:14.775168 8011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:14.776430 8011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:14.785729 8017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:14.785739 8019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:14.785979 8016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:14.786139 8011 server_base.cc:1034] running on GCE node
I20260926 16:46:14.786474 8011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:14.787070 8011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:14.788236 8011 hybrid_clock.cc:648] HybridClock initialized: now 1790441174788204 us; error 44 us; skew 500 ppm
I20260926 16:46:14.790184 8011 webserver.cc:460] Webserver started at http://127.21.12.66:34133/ using document root <none> and password file <none>
I20260926 16:46:14.790743 8011 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:14.790803 8011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:14.790997 8011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:46:14.793013 8011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-1/data/instance:
uuid: "e2c7982857474fd39c5f4bb7285109a8"
format_stamp: "Formatted at 2026-09-26 16:46:14 on dist-test-slave-n1hw"
server_key: "161055a829b95a824213bb36b5f032e9"
server_key_iv: "e6e38c88787df834df49594f39801152"
server_key_version: "encryptionkey@0"
I20260926 16:46:14.793689 8011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance:
uuid: "e2c7982857474fd39c5f4bb7285109a8"
format_stamp: "Formatted at 2026-09-26 16:46:14 on dist-test-slave-n1hw"
server_key: "161055a829b95a824213bb36b5f032e9"
server_key_iv: "e6e38c88787df834df49594f39801152"
server_key_version: "encryptionkey@0"
I20260926 16:46:14.797287 8011 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 16:46:14.799772 8025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:14.801002 8011 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:46:14.801129 8011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-1/wal
uuid: "e2c7982857474fd39c5f4bb7285109a8"
format_stamp: "Formatted at 2026-09-26 16:46:14 on dist-test-slave-n1hw"
server_key: "161055a829b95a824213bb36b5f032e9"
server_key_iv: "e6e38c88787df834df49594f39801152"
server_key_version: "encryptionkey@0"
I20260926 16:46:14.801236 8011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:14.814661 8011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:14.815281 8011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:14.815460 8011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:14.816002 8011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:46:14.816864 8011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:46:14.816931 8011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:14.817020 8011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:46:14.817066 8011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:14.825819 8011 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.66:42181
I20260926 16:46:14.825845 8138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.66:42181 every 8 connection(s)
I20260926 16:46:14.826668 8011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
I20260926 16:46:14.830895 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 8011
I20260926 16:46:14.831022 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance
I20260926 16:46:14.831349 21553 external_mini_cluster.cc:1428] Setting key 3c3a7f82039370a86839911c9fda18c3
I20260926 16:46:14.833217 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.67:0
--local_ip_for_outbound_sockets=127.21.12.67
--webserver_interface=127.21.12.67
--webserver_port=0
--tserver_master_addrs=127.21.12.126:38209
--builtin_ntp_servers=127.21.12.84:40647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 16:46:14.836159 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:14.828494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:34211 (local address 127.21.12.126:38209)
0926 16:46:14.828651 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:14.828656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:14.829731 (+ 1075us) server_negotiation.cc:407] Connection header received
0926 16:46:14.831135 (+ 1404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:14.831138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:14.831195 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:14.831280 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:14.832302 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:14.832931 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:14.834171 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:14.834386 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:14.834485 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:14.835655 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:14.835660 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:14.835665 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:46:14.835678 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:14.835703 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:46:14.835712 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:14.835787 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:14.835961 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 16:46:14.836020 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 16:46:14.837090 8141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:14.828970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38209 (local address 127.21.12.66:34211)
0926 16:46:14.829589 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:14.829627 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:14.830889 (+ 1262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:14.831514 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:14.831522 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:14.831607 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:46:14.832173 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:14.832188 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:14.833057 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:14.833061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:14.834046 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:14.834055 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:14.834529 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:14.834532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:14.834686 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:14.835362 (+ 676us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:46:14.835364 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:46:14.835400 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:46:14.835456 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:46:14.835796 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:46:14.835807 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:46:14.835817 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 16:46:14.835952 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 16:46:14.836226 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":113,"threads_started":1}
I20260926 16:46:14.838325 8139 heartbeater.cc:347] Connected to a master server at 127.21.12.126:38209
I20260926 16:46:14.838608 8139 heartbeater.cc:464] Registering TS with master...
I20260926 16:46:14.839124 8139 heartbeater.cc:511] Master 127.21.12.126:38209 requested a full tablet report, sending...
I20260926 16:46:14.840210 7818 ts_manager.cc:194] Registered new tserver with Master: e2c7982857474fd39c5f4bb7285109a8 (127.21.12.66:42181)
I20260926 16:46:14.841077 7818 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.12.66:34211
W20260926 16:46:14.942452 8142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:14.942705 8142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:14.942744 8142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:14.946640 8142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:14.946686 8142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:14.946739 8142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.67
I20260926 16:46:14.951398 8142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:40647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.67
--webserver_port=0
--tserver_master_addrs=127.21.12.126:38209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.67
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:46:14.952355 8142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:14.953635 8142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:14.962963 8147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:14.962952 8148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:14.962953 8150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:14.963245 8142 server_base.cc:1034] running on GCE node
I20260926 16:46:14.963716 8142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:14.964345 8142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:14.965502 8142 hybrid_clock.cc:648] HybridClock initialized: now 1790441174965464 us; error 49 us; skew 500 ppm
I20260926 16:46:14.967442 8142 webserver.cc:460] Webserver started at http://127.21.12.67:36337/ using document root <none> and password file <none>
I20260926 16:46:14.968019 8142 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:14.968082 8142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:14.968286 8142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:46:14.970484 8142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-2/data/instance:
uuid: "ea3efef946c641a689e5ff02b0d98a0d"
format_stamp: "Formatted at 2026-09-26 16:46:14 on dist-test-slave-n1hw"
server_key: "a8950c6ba91b6d49901bfad26881ad79"
server_key_iv: "d6e8f7d730b9ac6efa57b172430a2d25"
server_key_version: "encryptionkey@0"
I20260926 16:46:14.971060 8142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance:
uuid: "ea3efef946c641a689e5ff02b0d98a0d"
format_stamp: "Formatted at 2026-09-26 16:46:14 on dist-test-slave-n1hw"
server_key: "a8950c6ba91b6d49901bfad26881ad79"
server_key_iv: "d6e8f7d730b9ac6efa57b172430a2d25"
server_key_version: "encryptionkey@0"
I20260926 16:46:14.974792 8142 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:46:14.977362 8156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:14.978848 8142 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:46:14.978945 8142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-2/wal
uuid: "ea3efef946c641a689e5ff02b0d98a0d"
format_stamp: "Formatted at 2026-09-26 16:46:14 on dist-test-slave-n1hw"
server_key: "a8950c6ba91b6d49901bfad26881ad79"
server_key_iv: "d6e8f7d730b9ac6efa57b172430a2d25"
server_key_version: "encryptionkey@0"
I20260926 16:46:14.979087 8142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:14.991943 8142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:14.992581 8142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:14.992728 8142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:14.993257 8142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:46:14.994235 8142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:46:14.994287 8142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:14.994326 8142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:46:14.994342 8142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:15.003306 8142 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.67:33673
I20260926 16:46:15.003463 8271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.67:33673 every 8 connection(s)
I20260926 16:46:15.004438 8142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
I20260926 16:46:15.009634 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 8142
I20260926 16:46:15.009773 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance
I20260926 16:46:15.010077 21553 external_mini_cluster.cc:1428] Setting key 82bf264183314763ba31d0f842ab8753
I20260926 16:46:15.014235 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:15.006262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:35325 (local address 127.21.12.126:38209)
0926 16:46:15.006396 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:15.006400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:15.007435 (+ 1035us) server_negotiation.cc:407] Connection header received
0926 16:46:15.008835 (+ 1400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:15.008838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:15.008896 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:15.008992 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:15.010173 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:15.010901 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:15.012479 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:15.012703 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:15.012785 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:15.013839 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:15.013842 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:15.013844 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:46:15.013855 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:15.013874 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:46:15.013882 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:15.013941 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:15.014078 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 16:46:15.014130 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 16:46:15.015089 8274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:15.006650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:38209 (local address 127.21.12.67:35325)
0926 16:46:15.007297 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:15.007336 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:15.008621 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:15.009176 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:15.009183 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:15.009267 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:46:15.009958 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:15.009979 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:15.011049 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:15.011053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:15.012318 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:15.012328 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:15.012827 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:15.012830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:15.012970 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:15.013564 (+ 594us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:46:15.013566 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:46:15.013604 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:46:15.013666 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:46:15.013960 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:46:15.013967 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:46:15.013971 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 16:46:15.014067 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 16:46:15.014281 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":108,"threads_started":1}
I20260926 16:46:15.016242 8272 heartbeater.cc:347] Connected to a master server at 127.21.12.126:38209
I20260926 16:46:15.016484 8272 heartbeater.cc:464] Registering TS with master...
I20260926 16:46:15.016987 8272 heartbeater.cc:511] Master 127.21.12.126:38209 requested a full tablet report, sending...
I20260926 16:46:15.018026 7818 ts_manager.cc:194] Registered new tserver with Master: ea3efef946c641a689e5ff02b0d98a0d (127.21.12.67:33673)
I20260926 16:46:15.018553 7818 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.12.67:35325
I20260926 16:46:15.023936 21553 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:46:15.029310 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:15.025847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57660 (local address 127.21.12.126:38209)
0926 16:46:15.025979 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:15.025987 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:15.026146 (+ 159us) server_negotiation.cc:407] Connection header received
0926 16:46:15.026236 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:15.026238 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:15.026271 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:15.026339 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:15.026566 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:15.027110 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:15.027920 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:15.028133 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:15.028214 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:15.028947 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:15.028950 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:15.028952 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:46:15.028963 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:15.028978 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:46:15.028985 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:15.029080 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:15.029185 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 16:46:15.029215 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 16:46:15.032014 7818 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57660:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:46:15.035037 7818 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:46:15.048864 8286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:15.043875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:37805 (local address 127.0.0.1:42690)
0926 16:46:15.044377 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:15.044392 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:15.044473 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:15.044939 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:15.044942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:15.044960 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:46:15.045105 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:15.045112 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:15.046250 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:15.046256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:15.047316 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:15.047325 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:15.048348 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:15.048352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:15.048456 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:15.048494 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 16:46:15.048611 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 16:46:15.048719 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":86,"threads_started":1}
I20260926 16:46:15.049099 8283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:15.043153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:42181 (local address 127.0.0.1:45544)
0926 16:46:15.043641 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:15.043690 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:15.043817 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:15.044923 (+ 1106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:15.044929 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:15.044952 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:46:15.045104 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:15.045113 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:15.046535 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:15.046539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:15.047695 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:15.047704 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:15.048722 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:15.048726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:15.048811 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:15.048839 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 16:46:15.048887 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 16:46:15.048955 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":86,"threads_started":1}
I20260926 16:46:15.048913 8288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:15.044013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42690 (local address 127.21.12.65:37805)
0926 16:46:15.044469 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:15.044481 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:15.044505 (+ 24us) server_negotiation.cc:407] Connection header received
0926 16:46:15.044591 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:15.044596 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:15.044795 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:15.044964 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:46:15.045216 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:15.046151 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:15.047454 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:15.048228 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:15.048336 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:15.048529 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:15.048641 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 16:46:15.048744 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":104,"threads_started":1}
I20260926 16:46:15.049470 8285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:15.043505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.67:33673 (local address 127.0.0.1:44052)
0926 16:46:15.044063 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:15.044077 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:15.044158 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:15.045648 (+ 1490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:15.045651 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:15.045675 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:46:15.045776 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:15.045783 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:15.047074 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:15.047077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:15.048060 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:15.048067 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:15.049076 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:15.049082 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:15.049185 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:15.049212 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 16:46:15.049261 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 16:46:15.049335 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":104,"threads_started":1}
I20260926 16:46:15.049930 8284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:15.043330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45544 (local address 127.21.12.66:42181)
0926 16:46:15.044409 (+ 1079us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:15.044423 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:15.044448 (+ 25us) server_negotiation.cc:407] Connection header received
0926 16:46:15.044524 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:15.044529 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:15.044752 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:15.044923 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:46:15.045240 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:15.046404 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:15.047813 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:15.048590 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:15.049393 (+ 803us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:15.049611 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:15.049660 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 16:46:15.049742 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":974,"thread_start_us":133,"threads_started":1}
I20260926 16:46:15.050228 8287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:15.043921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44052 (local address 127.21.12.67:33673)
0926 16:46:15.045230 (+ 1309us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:15.045241 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:15.045261 (+ 20us) server_negotiation.cc:407] Connection header received
0926 16:46:15.045332 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:15.045336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:15.045500 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:15.045656 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:46:15.045876 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:15.046950 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:15.048167 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:15.048924 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:15.049730 (+ 806us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:15.049928 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:15.049974 (+ 46us) server_negotiation.cc:299] Negotiation successful
0926 16:46:15.050065 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1201,"thread_start_us":136,"threads_started":1}
I20260926 16:46:15.052184 7941 tablet_service.cc:1467] Processing CreateTablet for tablet 9feb62704b234c9a9471ed4fd65f8a28 (DEFAULT_TABLE table=test-table [id=b68fca63e1db44d2abea94c2dc1c97c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:46:15.052722 8205 tablet_service.cc:1467] Processing CreateTablet for tablet 9feb62704b234c9a9471ed4fd65f8a28 (DEFAULT_TABLE table=test-table [id=b68fca63e1db44d2abea94c2dc1c97c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:46:15.052716 8073 tablet_service.cc:1467] Processing CreateTablet for tablet 9feb62704b234c9a9471ed4fd65f8a28 (DEFAULT_TABLE table=test-table [id=b68fca63e1db44d2abea94c2dc1c97c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:46:15.053157 7941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9feb62704b234c9a9471ed4fd65f8a28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:15.053714 8205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9feb62704b234c9a9471ed4fd65f8a28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:15.053714 8073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9feb62704b234c9a9471ed4fd65f8a28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:15.059903 8289 tablet_bootstrap.cc:492] T 9feb62704b234c9a9471ed4fd65f8a28 P 9020895d8ce14cbb9dd3a67aedae0db0: Bootstrap starting.
I20260926 16:46:15.061219 8291 tablet_bootstrap.cc:492] T 9feb62704b234c9a9471ed4fd65f8a28 P ea3efef946c641a689e5ff02b0d98a0d: Bootstrap starting.
I20260926 16:46:15.061219 8290 tablet_bootstrap.cc:492] T 9feb62704b234c9a9471ed4fd65f8a28 P e2c7982857474fd39c5f4bb7285109a8: Bootstrap starting.
I20260926 16:46:15.062330 8289 tablet_bootstrap.cc:654] T 9feb62704b234c9a9471ed4fd65f8a28 P 9020895d8ce14cbb9dd3a67aedae0db0: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:15.063068 8289 log.cc:824] T 9feb62704b234c9a9471ed4fd65f8a28 P 9020895d8ce14cbb9dd3a67aedae0db0: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:15.063329 8290 tablet_bootstrap.cc:654] T 9feb62704b234c9a9471ed4fd65f8a28 P e2c7982857474fd39c5f4bb7285109a8: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:15.063328 8291 tablet_bootstrap.cc:654] T 9feb62704b234c9a9471ed4fd65f8a28 P ea3efef946c641a689e5ff02b0d98a0d: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:15.064149 8291 log.cc:824] T 9feb62704b234c9a9471ed4fd65f8a28 P ea3efef946c641a689e5ff02b0d98a0d: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:15.064149 8290 log.cc:824] T 9feb62704b234c9a9471ed4fd65f8a28 P e2c7982857474fd39c5f4bb7285109a8: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:15.065062 8289 tablet_bootstrap.cc:492] T 9feb62704b234c9a9471ed4fd65f8a28 P 9020895d8ce14cbb9dd3a67aedae0db0: No bootstrap required, opened a new log
I20260926 16:46:15.065251 8289 ts_tablet_manager.cc:1397] T 9feb62704b234c9a9471ed4fd65f8a28 P 9020895d8ce14cbb9dd3a67aedae0db0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 16:46:15.066045 8291 tablet_bootstrap.cc:492] T 9feb62704b234c9a9471ed4fd65f8a28 P ea3efef946c641a689e5ff02b0d98a0d: No bootstrap required, opened a new log
I20260926 16:46:15.066254 8291 ts_tablet_manager.cc:1397] T 9feb62704b234c9a9471ed4fd65f8a28 P ea3efef946c641a689e5ff02b0d98a0d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 16:46:15.066540 8290 tablet_bootstrap.cc:492] T 9feb62704b234c9a9471ed4fd65f8a28 P e2c7982857474fd39c5f4bb7285109a8: No bootstrap required, opened a new log
I20260926 16:46:15.066740 8290 ts_tablet_manager.cc:1397] T 9feb62704b234c9a9471ed4fd65f8a28 P e2c7982857474fd39c5f4bb7285109a8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 16:46:15.068568 8289 raft_consensus.cc:348] T 9feb62704b234c9a9471ed4fd65f8a28 P 9020895d8ce14cbb9dd3a67aedae0db0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c7982857474fd39c5f4bb7285109a8" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 42181 } } peers { permanent_uuid: "9020895d8ce14cbb9dd3a67aedae0db0" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 37805 } } peers { permanent_uuid: "ea3efef946c641a689e5ff02b0d98a0d" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33673 } }
I20260926 16:46:15.068804 8289 raft_consensus.cc:374] T 9feb62704b234c9a9471ed4fd65f8a28 P 9020895d8ce14cbb9dd3a67aedae0db0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:46:15.068854 8289 raft_consensus.cc:729] T 9feb62704b234c9a9471ed4fd65f8a28 P 9020895d8ce14cbb9dd3a67aedae0db0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9020895d8ce14cbb9dd3a67aedae0db0, State: Initialized, Role: FOLLOWER
I20260926 16:46:15.069267 8289 consensus_queue.cc:260] T 9feb62704b234c9a9471ed4fd65f8a28 P 9020895d8ce14cbb9dd3a67aedae0db0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c7982857474fd39c5f4bb7285109a8" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 42181 } } peers { permanent_uuid: "9020895d8ce14cbb9dd3a67aedae0db0" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 37805 } } peers { permanent_uuid: "ea3efef946c641a689e5ff02b0d98a0d" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33673 } }
I20260926 16:46:15.069919 8008 heartbeater.cc:503] Master 127.21.12.126:38209 was elected leader, sending a full tablet report...
I20260926 16:46:15.070272 8289 ts_tablet_manager.cc:1428] T 9feb62704b234c9a9471ed4fd65f8a28 P 9020895d8ce14cbb9dd3a67aedae0db0: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 16:46:15.070431 8291 raft_consensus.cc:348] T 9feb62704b234c9a9471ed4fd65f8a28 P ea3efef946c641a689e5ff02b0d98a0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c7982857474fd39c5f4bb7285109a8" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 42181 } } peers { permanent_uuid: "9020895d8ce14cbb9dd3a67aedae0db0" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 37805 } } peers { permanent_uuid: "ea3efef946c641a689e5ff02b0d98a0d" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33673 } }
I20260926 16:46:15.070431 8290 raft_consensus.cc:348] T 9feb62704b234c9a9471ed4fd65f8a28 P e2c7982857474fd39c5f4bb7285109a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c7982857474fd39c5f4bb7285109a8" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 42181 } } peers { permanent_uuid: "9020895d8ce14cbb9dd3a67aedae0db0" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 37805 } } peers { permanent_uuid: "ea3efef946c641a689e5ff02b0d98a0d" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33673 } }
I20260926 16:46:15.070663 8291 raft_consensus.cc:374] T 9feb62704b234c9a9471ed4fd65f8a28 P ea3efef946c641a689e5ff02b0d98a0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:46:15.070663 8290 raft_consensus.cc:374] T 9feb62704b234c9a9471ed4fd65f8a28 P e2c7982857474fd39c5f4bb7285109a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:46:15.070711 8290 raft_consensus.cc:729] T 9feb62704b234c9a9471ed4fd65f8a28 P e2c7982857474fd39c5f4bb7285109a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c7982857474fd39c5f4bb7285109a8, State: Initialized, Role: FOLLOWER
I20260926 16:46:15.070721 8291 raft_consensus.cc:729] T 9feb62704b234c9a9471ed4fd65f8a28 P ea3efef946c641a689e5ff02b0d98a0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea3efef946c641a689e5ff02b0d98a0d, State: Initialized, Role: FOLLOWER
I20260926 16:46:15.071156 8291 consensus_queue.cc:260] T 9feb62704b234c9a9471ed4fd65f8a28 P ea3efef946c641a689e5ff02b0d98a0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c7982857474fd39c5f4bb7285109a8" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 42181 } } peers { permanent_uuid: "9020895d8ce14cbb9dd3a67aedae0db0" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 37805 } } peers { permanent_uuid: "ea3efef946c641a689e5ff02b0d98a0d" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33673 } }
I20260926 16:46:15.071156 8290 consensus_queue.cc:260] T 9feb62704b234c9a9471ed4fd65f8a28 P e2c7982857474fd39c5f4bb7285109a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c7982857474fd39c5f4bb7285109a8" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 42181 } } peers { permanent_uuid: "9020895d8ce14cbb9dd3a67aedae0db0" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 37805 } } peers { permanent_uuid: "ea3efef946c641a689e5ff02b0d98a0d" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33673 } }
I20260926 16:46:15.072062 8290 ts_tablet_manager.cc:1428] T 9feb62704b234c9a9471ed4fd65f8a28 P e2c7982857474fd39c5f4bb7285109a8: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 16:46:15.072067 8291 ts_tablet_manager.cc:1428] T 9feb62704b234c9a9471ed4fd65f8a28 P ea3efef946c641a689e5ff02b0d98a0d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 16:46:15.072386 8139 heartbeater.cc:503] Master 127.21.12.126:38209 was elected leader, sending a full tablet report...
I20260926 16:46:15.073252 8272 heartbeater.cc:503] Master 127.21.12.126:38209 was elected leader, sending a full tablet report...
W20260926 16:46:15.077530 8140 tablet.cc:2369] T 9feb62704b234c9a9471ed4fd65f8a28 P e2c7982857474fd39c5f4bb7285109a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:46:15.126492 8296 raft_consensus.cc:482] T 9feb62704b234c9a9471ed4fd65f8a28 P e2c7982857474fd39c5f4bb7285109a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:46:15.126730 8296 raft_consensus.cc:504] T 9feb62704b234c9a9471ed4fd65f8a28 P e2c7982857474fd39c5f4bb7285109a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c7982857474fd39c5f4bb7285109a8" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 42181 } } peers { permanent_uuid: "9020895d8ce14cbb9dd3a67aedae0db0" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 37805 } } peers { permanent_uuid: "ea3efef946c641a689e5ff02b0d98a0d" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33673 } }
I20260926 16:46:15.128114 8296 leader_election.cc:288] T 9feb62704b234c9a9471ed4fd65f8a28 P e2c7982857474fd39c5f4bb7285109a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9020895d8ce14cbb9dd3a67aedae0db0 (127.21.12.65:37805), ea3efef946c641a689e5ff02b0d98a0d (127.21.12.67:33673)
I20260926 16:46:15.131543 8141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:15.128204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:37805 (local address 127.21.12.66:36083)
0926 16:46:15.128349 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:15.128365 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:15.128457 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:15.128807 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:15.128810 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:15.128831 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:46:15.128947 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:15.128954 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:15.129670 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:15.129674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:15.130496 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:15.130505 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:15.131132 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:15.131135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:15.131237 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:15.131255 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 16:46:15.131340 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 16:46:15.131409 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 16:46:15.131553 8288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:15.128288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:36083 (local address 127.21.12.65:37805)
0926 16:46:15.128494 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:15.128501 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:15.128537 (+ 36us) server_negotiation.cc:407] Connection header received
0926 16:46:15.128634 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:15.128637 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:15.128695 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:15.128791 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:46:15.129055 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:15.129561 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:15.130619 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:15.131002 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:15.131139 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:15.131189 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:15.131354 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 16:46:15.131422 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260926 16:46:15.132011 8287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:15.128531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:44847 (local address 127.21.12.67:33673)
0926 16:46:15.128652 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:15.128656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:15.128871 (+ 215us) server_negotiation.cc:407] Connection header received
0926 16:46:15.129031 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:15.129034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:15.129084 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:15.129164 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:46:15.129433 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:15.129985 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:15.130985 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:15.131376 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:15.131512 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:15.131557 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:15.131812 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 16:46:15.131892 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 16:46:15.132165 7961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9feb62704b234c9a9471ed4fd65f8a28" candidate_uuid: "e2c7982857474fd39c5f4bb7285109a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9020895d8ce14cbb9dd3a67aedae0db0" is_pre_election: true
I20260926 16:46:15.132383 8298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:15.128335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.67:33673 (local address 127.21.12.66:44847)
0926 16:46:15.128764 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:15.128780 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:15.128898 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:15.129177 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:15.129179 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:15.129194 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:46:15.129320 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:15.129326 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:15.130089 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:15.130094 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:15.130851 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:15.130860 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:15.131510 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:15.131513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:15.131612 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:15.131640 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 16:46:15.132172 (+ 532us) client_negotiation.cc:240] Negotiation successful
0926 16:46:15.132240 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":198,"threads_started":1}
I20260926 16:46:15.132493 7961 raft_consensus.cc:2454] T 9feb62704b234c9a9471ed4fd65f8a28 P 9020895d8ce14cbb9dd3a67aedae0db0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2c7982857474fd39c5f4bb7285109a8 in term 0.
I20260926 16:46:15.132992 8027 leader_election.cc:302] T 9feb62704b234c9a9471ed4fd65f8a28 P e2c7982857474fd39c5f4bb7285109a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9020895d8ce14cbb9dd3a67aedae0db0, e2c7982857474fd39c5f4bb7285109a8; no voters:
I20260926 16:46:15.133008 8225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9feb62704b234c9a9471ed4fd65f8a28" candidate_uuid: "e2c7982857474fd39c5f4bb7285109a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea3efef946c641a689e5ff02b0d98a0d" is_pre_election: true
I20260926 16:46:15.133270 8225 raft_consensus.cc:2454] T 9feb62704b234c9a9471ed4fd65f8a28 P ea3efef946c641a689e5ff02b0d98a0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2c7982857474fd39c5f4bb7285109a8 in term 0.
I20260926 16:46:15.133270 8296 raft_consensus.cc:2781] T 9feb62704b234c9a9471ed4fd65f8a28 P e2c7982857474fd39c5f4bb7285109a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:46:15.133359 8296 raft_consensus.cc:482] T 9feb62704b234c9a9471ed4fd65f8a28 P e2c7982857474fd39c5f4bb7285109a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:46:15.133419 8296 raft_consensus.cc:3037] T 9feb62704b234c9a9471ed4fd65f8a28 P e2c7982857474fd39c5f4bb7285109a8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:15.134591 8296 raft_consensus.cc:504] T 9feb62704b234c9a9471ed4fd65f8a28 P e2c7982857474fd39c5f4bb7285109a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c7982857474fd39c5f4bb7285109a8" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 42181 } } peers { permanent_uuid: "9020895d8ce14cbb9dd3a67aedae0db0" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 37805 } } peers { permanent_uuid: "ea3efef946c641a689e5ff02b0d98a0d" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33673 } }
I20260926 16:46:15.134979 8296 leader_election.cc:288] T 9feb62704b234c9a9471ed4fd65f8a28 P e2c7982857474fd39c5f4bb7285109a8 [CANDIDATE]: Term 1 election: Requested vote from peers 9020895d8ce14cbb9dd3a67aedae0db0 (127.21.12.65:37805), ea3efef946c641a689e5ff02b0d98a0d (127.21.12.67:33673)
I20260926 16:46:15.135345 8225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9feb62704b234c9a9471ed4fd65f8a28" candidate_uuid: "e2c7982857474fd39c5f4bb7285109a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea3efef946c641a689e5ff02b0d98a0d"
I20260926 16:46:15.135315 7961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9feb62704b234c9a9471ed4fd65f8a28" candidate_uuid: "e2c7982857474fd39c5f4bb7285109a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9020895d8ce14cbb9dd3a67aedae0db0"
I20260926 16:46:15.135519 7961 raft_consensus.cc:3037] T 9feb62704b234c9a9471ed4fd65f8a28 P 9020895d8ce14cbb9dd3a67aedae0db0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:15.135527 8225 raft_consensus.cc:3037] T 9feb62704b234c9a9471ed4fd65f8a28 P ea3efef946c641a689e5ff02b0d98a0d [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:15.136673 7961 raft_consensus.cc:2454] T 9feb62704b234c9a9471ed4fd65f8a28 P 9020895d8ce14cbb9dd3a67aedae0db0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2c7982857474fd39c5f4bb7285109a8 in term 1.
I20260926 16:46:15.136694 8225 raft_consensus.cc:2454] T 9feb62704b234c9a9471ed4fd65f8a28 P ea3efef946c641a689e5ff02b0d98a0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2c7982857474fd39c5f4bb7285109a8 in term 1.
I20260926 16:46:15.136945 8027 leader_election.cc:302] T 9feb62704b234c9a9471ed4fd65f8a28 P e2c7982857474fd39c5f4bb7285109a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9020895d8ce14cbb9dd3a67aedae0db0, e2c7982857474fd39c5f4bb7285109a8; no voters:
I20260926 16:46:15.137122 8296 raft_consensus.cc:2781] T 9feb62704b234c9a9471ed4fd65f8a28 P e2c7982857474fd39c5f4bb7285109a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:46:15.137354 8296 raft_consensus.cc:686] T 9feb62704b234c9a9471ed4fd65f8a28 P e2c7982857474fd39c5f4bb7285109a8 [term 1 LEADER]: Becoming Leader. State: Replica: e2c7982857474fd39c5f4bb7285109a8, State: Running, Role: LEADER
I20260926 16:46:15.137688 8296 consensus_queue.cc:237] T 9feb62704b234c9a9471ed4fd65f8a28 P e2c7982857474fd39c5f4bb7285109a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c7982857474fd39c5f4bb7285109a8" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 42181 } } peers { permanent_uuid: "9020895d8ce14cbb9dd3a67aedae0db0" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 37805 } } peers { permanent_uuid: "ea3efef946c641a689e5ff02b0d98a0d" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33673 } }
I20260926 16:46:15.141095 7817 catalog_manager.cc:5495] T 9feb62704b234c9a9471ed4fd65f8a28 P e2c7982857474fd39c5f4bb7285109a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2c7982857474fd39c5f4bb7285109a8 (127.21.12.66). New cstate: current_term: 1 leader_uuid: "e2c7982857474fd39c5f4bb7285109a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c7982857474fd39c5f4bb7285109a8" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 42181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9020895d8ce14cbb9dd3a67aedae0db0" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 37805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea3efef946c641a689e5ff02b0d98a0d" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33673 } health_report { overall_health: UNKNOWN } } }
W20260926 16:46:15.151789 8009 tablet.cc:2369] T 9feb62704b234c9a9471ed4fd65f8a28 P 9020895d8ce14cbb9dd3a67aedae0db0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:46:15.173818 8284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:15.170385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45558 (local address 127.21.12.66:42181)
0926 16:46:15.170507 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:15.170510 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:15.170523 (+ 13us) server_negotiation.cc:407] Connection header received
0926 16:46:15.170617 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:15.170619 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:15.170681 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:15.170769 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:46:15.170979 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:15.171472 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:15.172354 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:15.172578 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:15.172657 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:15.172943 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:46:15.173336 (+ 393us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:46:15.173443 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:15.173607 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 16:46:15.173643 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 16:46:15.179630 7961 raft_consensus.cc:1261] T 9feb62704b234c9a9471ed4fd65f8a28 P 9020895d8ce14cbb9dd3a67aedae0db0 [term 1 FOLLOWER]: Refusing update from remote peer e2c7982857474fd39c5f4bb7285109a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:46:15.179628 8225 raft_consensus.cc:1261] T 9feb62704b234c9a9471ed4fd65f8a28 P ea3efef946c641a689e5ff02b0d98a0d [term 1 FOLLOWER]: Refusing update from remote peer e2c7982857474fd39c5f4bb7285109a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:46:15.180380 8299 consensus_queue.cc:1035] T 9feb62704b234c9a9471ed4fd65f8a28 P e2c7982857474fd39c5f4bb7285109a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9020895d8ce14cbb9dd3a67aedae0db0" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 37805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:46:15.180555 8296 consensus_queue.cc:1035] T 9feb62704b234c9a9471ed4fd65f8a28 P e2c7982857474fd39c5f4bb7285109a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea3efef946c641a689e5ff02b0d98a0d" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:46:15.187734 8302 mvcc.cc:205] Tried to move back new op lower bound from 7333647053528932352 to 7333647053366771712. Current Snapshot: MvccSnapshot[applied={T|T < 7333647053528932352}]
I20260926 16:46:15.190025 8303 mvcc.cc:205] Tried to move back new op lower bound from 7333647053528932352 to 7333647053366771712. Current Snapshot: MvccSnapshot[applied={T|T < 7333647053528932352 or (T in {7333647053528932352})}]
I20260926 16:46:15.191417 8304 mvcc.cc:205] Tried to move back new op lower bound from 7333647053528932352 to 7333647053366771712. Current Snapshot: MvccSnapshot[applied={T|T < 7333647053528932352}]
I20260926 16:46:15.198701 7818 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:46:15.198844 7818 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:46:15.201145 7818 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 40d76984fa934938a0bd558587c45bea: Sending DeleteTablet for 3 replicas of tablet 9feb62704b234c9a9471ed4fd65f8a28
I20260926 16:46:15.201999 8205 tablet_service.cc:1514] Processing DeleteTablet for tablet 9feb62704b234c9a9471ed4fd65f8a28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:46:15 UTC) from {username='slave'} at 127.0.0.1:44052
I20260926 16:46:15.202003 7941 tablet_service.cc:1514] Processing DeleteTablet for tablet 9feb62704b234c9a9471ed4fd65f8a28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:46:15 UTC) from {username='slave'} at 127.0.0.1:42690
I20260926 16:46:15.202085 8073 tablet_service.cc:1514] Processing DeleteTablet for tablet 9feb62704b234c9a9471ed4fd65f8a28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:46:15 UTC) from {username='slave'} at 127.0.0.1:45544
I20260926 16:46:15.202474 8311 tablet_replica.cc:330] T 9feb62704b234c9a9471ed4fd65f8a28 P ea3efef946c641a689e5ff02b0d98a0d: stopping tablet replica
I20260926 16:46:15.202656 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 7879
I20260926 16:46:15.202802 8311 raft_consensus.cc:2229] T 9feb62704b234c9a9471ed4fd65f8a28 P ea3efef946c641a689e5ff02b0d98a0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:46:15.203171 8311 raft_consensus.cc:2258] T 9feb62704b234c9a9471ed4fd65f8a28 P ea3efef946c641a689e5ff02b0d98a0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:46:15.202664 8313 tablet_replica.cc:330] T 9feb62704b234c9a9471ed4fd65f8a28 P e2c7982857474fd39c5f4bb7285109a8: stopping tablet replica
I20260926 16:46:15.203842 8313 raft_consensus.cc:2229] T 9feb62704b234c9a9471ed4fd65f8a28 P e2c7982857474fd39c5f4bb7285109a8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:46:15.204448 8313 raft_consensus.cc:2258] T 9feb62704b234c9a9471ed4fd65f8a28 P e2c7982857474fd39c5f4bb7285109a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:46:15.205529 8311 ts_tablet_manager.cc:1905] T 9feb62704b234c9a9471ed4fd65f8a28 P ea3efef946c641a689e5ff02b0d98a0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:46:15.207165 8313 ts_tablet_manager.cc:1905] T 9feb62704b234c9a9471ed4fd65f8a28 P e2c7982857474fd39c5f4bb7285109a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:46:15.209055 8311 ts_tablet_manager.cc:1918] T 9feb62704b234c9a9471ed4fd65f8a28 P ea3efef946c641a689e5ff02b0d98a0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:46:15.209147 8311 log.cc:1196] T 9feb62704b234c9a9471ed4fd65f8a28 P ea3efef946c641a689e5ff02b0d98a0d: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-2/wal/wals/9feb62704b234c9a9471ed4fd65f8a28
I20260926 16:46:15.209457 8311 ts_tablet_manager.cc:1939] T 9feb62704b234c9a9471ed4fd65f8a28 P ea3efef946c641a689e5ff02b0d98a0d: Deleting consensus metadata
I20260926 16:46:15.209836 8313 ts_tablet_manager.cc:1918] T 9feb62704b234c9a9471ed4fd65f8a28 P e2c7982857474fd39c5f4bb7285109a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:46:15.209914 8313 log.cc:1196] T 9feb62704b234c9a9471ed4fd65f8a28 P e2c7982857474fd39c5f4bb7285109a8: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440924937351-21553-0/minicluster-data/ts-1/wal/wals/9feb62704b234c9a9471ed4fd65f8a28
I20260926 16:46:15.210201 8313 ts_tablet_manager.cc:1939] T 9feb62704b234c9a9471ed4fd65f8a28 P e2c7982857474fd39c5f4bb7285109a8: Deleting consensus metadata
I20260926 16:46:15.210784 7804 catalog_manager.cc:4841] TS ea3efef946c641a689e5ff02b0d98a0d (127.21.12.67:33673): tablet 9feb62704b234c9a9471ed4fd65f8a28 (table test-table [id=b68fca63e1db44d2abea94c2dc1c97c9]) successfully deleted
I20260926 16:46:15.211231 7803 catalog_manager.cc:4841] TS e2c7982857474fd39c5f4bb7285109a8 (127.21.12.66:42181): tablet 9feb62704b234c9a9471ed4fd65f8a28 (table test-table [id=b68fca63e1db44d2abea94c2dc1c97c9]) successfully deleted
W20260926 16:46:15.211493 7803 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.12.65:37805 (error 108)
I20260926 16:46:15.211649 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 8011
I20260926 16:46:15.213200 8285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:15.212855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:37805 (local address 127.0.0.1:42704)
0926 16:46:15.213009 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:15.213098 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.65:37805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 16:46:15.213361 7803 catalog_manager.cc:4584] TS 9020895d8ce14cbb9dd3a67aedae0db0 (127.21.12.65:37805): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9feb62704b234c9a9471ed4fd65f8a28: Network error: Client connection negotiation failed: client connection to 127.21.12.65:37805: connect: Connection refused (error 111)
I20260926 16:46:15.216864 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 8142
I20260926 16:46:15.222421 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 7787
2026-09-26T16:46:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1016 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:46:15 dist-test-slave-n1hw krb5kdc[8317](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:46:15 dist-test-slave-n1hw krb5kdc[8317](info): set up 2 sockets
Sep 26 16:46:15 dist-test-slave-n1hw krb5kdc[8317](info): commencing operation
krb5kdc: starting...
W20260926 16:46:16.259861 21553 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Sep 26 16:46:16 dist-test-slave-n1hw krb5kdc[8317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:46:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:46:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
I20260926 16:46:16.384981 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:40981
--webserver_interface=127.21.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:33163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:40981
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:46:16.493081 8330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:16.493337 8330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:16.493412 8330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:16.497366 8330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:46:16.497419 8330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:16.497439 8330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:16.497455 8330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:46:16.497509 8330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:46:16.502560 8330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:33163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:40981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:40981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:46:16.503545 8330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:16.504544 8330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:16.512955 8338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:16.512992 8335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:16.513176 8330 server_base.cc:1034] running on GCE node
W20260926 16:46:16.513005 8336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:16.513798 8330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:16.514837 8330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:16.516029 8330 hybrid_clock.cc:648] HybridClock initialized: now 1790441176516004 us; error 36 us; skew 500 ppm
Sep 26 16:46:16 dist-test-slave-n1hw krb5kdc[8317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441176, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:46:16.518744 8330 init.cc:377] Logged in from keytab as kudu/127.21.12.126@KRBTEST.COM (short username kudu)
I20260926 16:46:16.519548 8330 webserver.cc:460] Webserver started at http://127.21.12.126:34163/ using document root <none> and password file <none>
I20260926 16:46:16.520128 8330 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:16.520212 8330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:16.520426 8330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:46:16.522504 8330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "bed454a2f8964c8692d424b730206f54"
format_stamp: "Formatted at 2026-09-26 16:46:16 on dist-test-slave-n1hw"
server_key: "7fcc87a835211845b8b55d8d7893b7ab"
server_key_iv: "b0930931004dea23bfd9cd9899cf24dc"
server_key_version: "encryptionkey@0"
I20260926 16:46:16.523051 8330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "bed454a2f8964c8692d424b730206f54"
format_stamp: "Formatted at 2026-09-26 16:46:16 on dist-test-slave-n1hw"
server_key: "7fcc87a835211845b8b55d8d7893b7ab"
server_key_iv: "b0930931004dea23bfd9cd9899cf24dc"
server_key_version: "encryptionkey@0"
I20260926 16:46:16.526615 8330 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:46:16.528857 8346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:16.530045 8330 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:46:16.530174 8330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "bed454a2f8964c8692d424b730206f54"
format_stamp: "Formatted at 2026-09-26 16:46:16 on dist-test-slave-n1hw"
server_key: "7fcc87a835211845b8b55d8d7893b7ab"
server_key_iv: "b0930931004dea23bfd9cd9899cf24dc"
server_key_version: "encryptionkey@0"
I20260926 16:46:16.530277 8330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:16.551932 8330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:16.554879 8330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:16.555078 8330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:16.562247 8330 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.126:40981
I20260926 16:46:16.562270 8398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.126:40981 every 8 connection(s)
I20260926 16:46:16.563164 8330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:46:16.565896 8399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:16.570926 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 8330
I20260926 16:46:16.571058 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:46:16.571386 21553 external_mini_cluster.cc:1428] Setting key 55e6ad821f0b326f929f77a752b99d81
I20260926 16:46:16.571364 8399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bed454a2f8964c8692d424b730206f54: Bootstrap starting.
I20260926 16:46:16.573890 8399 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bed454a2f8964c8692d424b730206f54: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:16.574602 8399 log.cc:824] T 00000000000000000000000000000000 P bed454a2f8964c8692d424b730206f54: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:16.576372 8399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bed454a2f8964c8692d424b730206f54: No bootstrap required, opened a new log
Sep 26 16:46:16 dist-test-slave-n1hw krb5kdc[8317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:46:16.580817 8399 raft_consensus.cc:348] T 00000000000000000000000000000000 P bed454a2f8964c8692d424b730206f54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bed454a2f8964c8692d424b730206f54" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 40981 } }
I20260926 16:46:16.581059 8399 raft_consensus.cc:374] T 00000000000000000000000000000000 P bed454a2f8964c8692d424b730206f54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:46:16.581132 8399 raft_consensus.cc:729] T 00000000000000000000000000000000 P bed454a2f8964c8692d424b730206f54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bed454a2f8964c8692d424b730206f54, State: Initialized, Role: FOLLOWER
I20260926 16:46:16.581589 8399 consensus_queue.cc:260] T 00000000000000000000000000000000 P bed454a2f8964c8692d424b730206f54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bed454a2f8964c8692d424b730206f54" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 40981 } }
I20260926 16:46:16.581736 8399 raft_consensus.cc:388] T 00000000000000000000000000000000 P bed454a2f8964c8692d424b730206f54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:46:16.581823 8399 raft_consensus.cc:482] T 00000000000000000000000000000000 P bed454a2f8964c8692d424b730206f54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:46:16.581928 8399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bed454a2f8964c8692d424b730206f54 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:16.582799 8399 raft_consensus.cc:504] T 00000000000000000000000000000000 P bed454a2f8964c8692d424b730206f54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bed454a2f8964c8692d424b730206f54" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 40981 } }
I20260926 16:46:16.583139 8399 leader_election.cc:302] T 00000000000000000000000000000000 P bed454a2f8964c8692d424b730206f54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bed454a2f8964c8692d424b730206f54; no voters:
I20260926 16:46:16.583433 8399 leader_election.cc:288] T 00000000000000000000000000000000 P bed454a2f8964c8692d424b730206f54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:46:16.583570 8404 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bed454a2f8964c8692d424b730206f54 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:46:16.583784 8404 raft_consensus.cc:686] T 00000000000000000000000000000000 P bed454a2f8964c8692d424b730206f54 [term 1 LEADER]: Becoming Leader. State: Replica: bed454a2f8964c8692d424b730206f54, State: Running, Role: LEADER
I20260926 16:46:16.584187 8404 consensus_queue.cc:237] T 00000000000000000000000000000000 P bed454a2f8964c8692d424b730206f54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bed454a2f8964c8692d424b730206f54" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 40981 } }
I20260926 16:46:16.584743 8399 sys_catalog.cc:564] T 00000000000000000000000000000000 P bed454a2f8964c8692d424b730206f54 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:46:16.586591 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:16.572777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43218 (local address 127.21.12.126:40981)
0926 16:46:16.573229 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:16.573244 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:16.573278 (+ 34us) server_negotiation.cc:407] Connection header received
0926 16:46:16.573982 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:16.574010 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:16.574341 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:16.574692 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:16.575482 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:16.576435 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:16.577587 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:16.578099 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:16.578213 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:16.581014 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:16.581020 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:16.581032 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:16.581083 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:16.583060 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:16.583589 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:16.583597 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:16.583603 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:16.583758 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:16.584075 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:16.584079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:16.584081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:16.584587 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:16.584805 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:16.585385 (+ 580us) server_negotiation.cc:299] Negotiation successful
0926 16:46:16.585706 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":127,"threads_started":1}
I20260926 16:46:16.587879 8405 sys_catalog.cc:455] T 00000000000000000000000000000000 P bed454a2f8964c8692d424b730206f54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bed454a2f8964c8692d424b730206f54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bed454a2f8964c8692d424b730206f54" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 40981 } } }
I20260926 16:46:16.588006 8405 sys_catalog.cc:458] T 00000000000000000000000000000000 P bed454a2f8964c8692d424b730206f54 [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:16.588266 8406 sys_catalog.cc:455] T 00000000000000000000000000000000 P bed454a2f8964c8692d424b730206f54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bed454a2f8964c8692d424b730206f54. Latest consensus state: current_term: 1 leader_uuid: "bed454a2f8964c8692d424b730206f54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bed454a2f8964c8692d424b730206f54" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 40981 } } }
I20260926 16:46:16.588420 8406 sys_catalog.cc:458] T 00000000000000000000000000000000 P bed454a2f8964c8692d424b730206f54 [sys.catalog]: This master's current role is: LEADER
W20260926 16:46:16.589939 8418 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bed454a2f8964c8692d424b730206f54: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 16:46:16.589998 8418 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 16:46:16.590080 8420 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:46:16.590974 8420 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:46:16.596681 8420 catalog_manager.cc:1347] Generated new cluster ID: 6048591b5f954228825bc8111f0e38dc
I20260926 16:46:16.596757 8420 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:46:16.607549 8420 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:46:16.608562 8420 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:46:16.620096 8420 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bed454a2f8964c8692d424b730206f54: Generated new TSK 0
I20260926 16:46:16.620780 8420 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab.
Entry for principal kudu/127.21.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab.
I20260926 16:46:16.657115 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.65:0
--local_ip_for_outbound_sockets=127.21.12.65
--webserver_interface=127.21.12.65
--webserver_port=0
--tserver_master_addrs=127.21.12.126:40981
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab
--principal=kudu/127.21.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:33163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:46:16.788367 8425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:16.788657 8425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:16.788727 8425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:16.792953 8425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:16.793059 8425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:16.793167 8425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.65
I20260926 16:46:16.798708 8425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:33163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.65.keytab
--principal=kudu/127.21.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.65
--webserver_port=0
--tserver_master_addrs=127.21.12.126:40981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.65
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:46:16.799682 8425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:16.800635 8425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:16.810321 8430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:16.810369 8431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:16.810382 8433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:16.810487 8425 server_base.cc:1034] running on GCE node
I20260926 16:46:16.810964 8425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:16.811591 8425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:16.812783 8425 hybrid_clock.cc:648] HybridClock initialized: now 1790441176812764 us; error 28 us; skew 500 ppm
Sep 26 16:46:16 dist-test-slave-n1hw krb5kdc[8317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441176, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:46:16.816648 8425 init.cc:377] Logged in from keytab as kudu/127.21.12.65@KRBTEST.COM (short username kudu)
I20260926 16:46:16.817880 8425 webserver.cc:460] Webserver started at http://127.21.12.65:35301/ using document root <none> and password file <none>
I20260926 16:46:16.818665 8425 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:16.818759 8425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:16.819021 8425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:46:16.821583 8425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-0/data/instance:
uuid: "398bb08e3d5b43f28620ac1d5577da35"
format_stamp: "Formatted at 2026-09-26 16:46:16 on dist-test-slave-n1hw"
server_key: "5d6498e285f8fc0f57d3b04911f6a122"
server_key_iv: "989f8a8806ca96bef02e322e1b67dd18"
server_key_version: "encryptionkey@0"
I20260926 16:46:16.822156 8425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance:
uuid: "398bb08e3d5b43f28620ac1d5577da35"
format_stamp: "Formatted at 2026-09-26 16:46:16 on dist-test-slave-n1hw"
server_key: "5d6498e285f8fc0f57d3b04911f6a122"
server_key_iv: "989f8a8806ca96bef02e322e1b67dd18"
server_key_version: "encryptionkey@0"
I20260926 16:46:16.825717 8425 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:46:16.828176 8440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:16.829404 8425 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:46:16.829560 8425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-0/wal
uuid: "398bb08e3d5b43f28620ac1d5577da35"
format_stamp: "Formatted at 2026-09-26 16:46:16 on dist-test-slave-n1hw"
server_key: "5d6498e285f8fc0f57d3b04911f6a122"
server_key_iv: "989f8a8806ca96bef02e322e1b67dd18"
server_key_version: "encryptionkey@0"
I20260926 16:46:16.829711 8425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:16.847949 8425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:16.850998 8425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:16.851188 8425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:16.851729 8425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:46:16.852504 8425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:46:16.852576 8425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:16.852655 8425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:46:16.852702 8425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:16.861152 8425 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.65:43795
I20260926 16:46:16.861174 8554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.65:43795 every 8 connection(s)
I20260926 16:46:16.862103 8425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
I20260926 16:46:16.864166 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 8425
I20260926 16:46:16.864302 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance
I20260926 16:46:16.864630 21553 external_mini_cluster.cc:1428] Setting key 774eb2c8afd2d6257df99a633bdc8b08
Sep 26 16:46:16 dist-test-slave-n1hw krb5kdc[8317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441176, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.65@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.12.66@KRBTEST.COM; defaulting to no policy
I20260926 16:46:16.875825 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:16.863854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:35959 (local address 127.21.12.126:40981)
0926 16:46:16.864030 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:16.864035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:16.865313 (+ 1278us) server_negotiation.cc:407] Connection header received
0926 16:46:16.866309 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:16.866312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:16.866362 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:16.866440 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:16.867565 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:16.868087 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:16.869225 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:16.869469 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:16.869596 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:16.872076 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:16.872079 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:16.872082 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:16.872105 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:16.873730 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:16.874267 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:16.874272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:16.874274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:16.874361 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:16.874700 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:16.874705 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:16.874707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:16.874884 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:16.874992 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:16.875530 (+ 538us) server_negotiation.cc:299] Negotiation successful
0926 16:46:16.875653 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.66@KRBTEST.COM" created.
I20260926 16:46:16.876704 8557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:16.864226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:40981 (local address 127.21.12.65:35959)
0926 16:46:16.865170 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:16.865221 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:16.866118 (+ 897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:16.866617 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:16.866624 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:16.866963 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:46:16.867421 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:16.867433 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:16.868206 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:16.868210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:16.869105 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:16.869114 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:16.869643 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:16.869647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:16.869877 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:16.870513 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:46:16.870528 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:46:16.871916 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:46:16.873874 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:16.873880 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:16.873895 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:16.874165 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:16.874454 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:16.874457 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:16.874459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:16.874607 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:16.874995 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:46:16.875004 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:46:16.875294 (+ 290us) client_negotiation.cc:769] Sending connection context
0926 16:46:16.875482 (+ 188us) client_negotiation.cc:240] Negotiation successful
0926 16:46:16.875854 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"thread_start_us":592,"threads_started":1}
I20260926 16:46:16.878153 8555 heartbeater.cc:347] Connected to a master server at 127.21.12.126:40981
I20260926 16:46:16.878391 8555 heartbeater.cc:464] Registering TS with master...
I20260926 16:46:16.879014 8555 heartbeater.cc:511] Master 127.21.12.126:40981 requested a full tablet report, sending...
I20260926 16:46:16.880679 8363 ts_manager.cc:194] Registered new tserver with Master: 398bb08e3d5b43f28620ac1d5577da35 (127.21.12.65:43795)
I20260926 16:46:16.881831 8363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.65@KRBTEST.COM'} at 127.21.12.65:35959
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab.
Entry for principal kudu/127.21.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab.
I20260926 16:46:16.894006 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.66:0
--local_ip_for_outbound_sockets=127.21.12.66
--webserver_interface=127.21.12.66
--webserver_port=0
--tserver_master_addrs=127.21.12.126:40981
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab
--principal=kudu/127.21.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:33163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:46:17.004302 8560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:17.004554 8560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:17.004596 8560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:17.008648 8560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:17.008699 8560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:17.008785 8560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.66
I20260926 16:46:17.013962 8560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:33163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.66.keytab
--principal=kudu/127.21.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.66
--webserver_port=0
--tserver_master_addrs=127.21.12.126:40981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.66
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:46:17.014935 8560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:17.015906 8560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:17.025316 8566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:17.025349 8565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:17.025336 8568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:17.025673 8560 server_base.cc:1034] running on GCE node
I20260926 16:46:17.026126 8560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:17.026849 8560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:17.028034 8560 hybrid_clock.cc:648] HybridClock initialized: now 1790441177027998 us; error 56 us; skew 500 ppm
Sep 26 16:46:17 dist-test-slave-n1hw krb5kdc[8317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441177, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:46:17.030870 8560 init.cc:377] Logged in from keytab as kudu/127.21.12.66@KRBTEST.COM (short username kudu)
I20260926 16:46:17.031693 8560 webserver.cc:460] Webserver started at http://127.21.12.66:46737/ using document root <none> and password file <none>
I20260926 16:46:17.032248 8560 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:17.032333 8560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:17.032631 8560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:46:17.034909 8560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-1/data/instance:
uuid: "432c67dd5fe34190b781f112b3985bab"
format_stamp: "Formatted at 2026-09-26 16:46:17 on dist-test-slave-n1hw"
server_key: "68f66c4839dab3bad753eeeec76f2aaa"
server_key_iv: "0c3b08bac7f2b30df1924dca490aad30"
server_key_version: "encryptionkey@0"
I20260926 16:46:17.035478 8560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance:
uuid: "432c67dd5fe34190b781f112b3985bab"
format_stamp: "Formatted at 2026-09-26 16:46:17 on dist-test-slave-n1hw"
server_key: "68f66c4839dab3bad753eeeec76f2aaa"
server_key_iv: "0c3b08bac7f2b30df1924dca490aad30"
server_key_version: "encryptionkey@0"
I20260926 16:46:17.039144 8560 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:46:17.041589 8575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:17.042837 8560 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:46:17.042974 8560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-1/wal
uuid: "432c67dd5fe34190b781f112b3985bab"
format_stamp: "Formatted at 2026-09-26 16:46:17 on dist-test-slave-n1hw"
server_key: "68f66c4839dab3bad753eeeec76f2aaa"
server_key_iv: "0c3b08bac7f2b30df1924dca490aad30"
server_key_version: "encryptionkey@0"
I20260926 16:46:17.043082 8560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:17.071028 8560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:17.074199 8560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:17.074399 8560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:17.074975 8560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:46:17.075807 8560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:46:17.075874 8560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:17.075941 8560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:46:17.075986 8560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:17.084650 8560 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.66:43311
I20260926 16:46:17.084708 8689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.66:43311 every 8 connection(s)
I20260926 16:46:17.085706 8560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
I20260926 16:46:17.090741 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 8560
I20260926 16:46:17.090862 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance
I20260926 16:46:17.091156 21553 external_mini_cluster.cc:1428] Setting key 42dc466213f09990fd79c4c4ed450080
Sep 26 16:46:17 dist-test-slave-n1hw krb5kdc[8317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441177, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.66@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.12.67@KRBTEST.COM; defaulting to no policy
I20260926 16:46:17.100581 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:17.087548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:60719 (local address 127.21.12.126:40981)
0926 16:46:17.087679 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:17.087683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:17.088545 (+ 862us) server_negotiation.cc:407] Connection header received
0926 16:46:17.089942 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:17.089946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:17.090004 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:17.090104 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:17.091490 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:17.092131 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:17.093387 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:17.093809 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:17.093931 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:17.096614 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:17.096619 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:17.096622 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:17.096657 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:17.098397 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:17.098954 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:17.098960 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:17.098962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:17.099058 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:17.099438 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:17.099443 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:17.099445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:17.099647 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:17.099756 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:17.100271 (+ 515us) server_negotiation.cc:299] Negotiation successful
0926 16:46:17.100397 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 16:46:17.101456 8692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:17.087925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:40981 (local address 127.21.12.66:60719)
0926 16:46:17.088398 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:17.088455 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:17.089685 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:17.090297 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:17.090304 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:17.090753 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:46:17.091353 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:17.091368 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:17.092277 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:17.092282 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:17.093261 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:17.093271 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:17.093953 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:17.093956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:17.094100 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:17.094743 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:46:17.094760 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:46:17.096438 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:46:17.098529 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:17.098535 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:17.098550 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:17.098832 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:17.099161 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:17.099164 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:17.099166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:17.099315 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:17.099768 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:46:17.099777 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:46:17.100026 (+ 249us) client_negotiation.cc:769] Sending connection context
0926 16:46:17.100230 (+ 204us) client_negotiation.cc:240] Negotiation successful
0926 16:46:17.100581 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":113,"threads_started":1}
I20260926 16:46:17.102737 8690 heartbeater.cc:347] Connected to a master server at 127.21.12.126:40981
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.67@KRBTEST.COM" created.
I20260926 16:46:17.103009 8690 heartbeater.cc:464] Registering TS with master...
I20260926 16:46:17.103513 8690 heartbeater.cc:511] Master 127.21.12.126:40981 requested a full tablet report, sending...
I20260926 16:46:17.104635 8363 ts_manager.cc:194] Registered new tserver with Master: 432c67dd5fe34190b781f112b3985bab (127.21.12.66:43311)
I20260926 16:46:17.105201 8363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.66@KRBTEST.COM'} at 127.21.12.66:60719
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab.
Entry for principal kudu/127.21.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab.
I20260926 16:46:17.121606 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.67:0
--local_ip_for_outbound_sockets=127.21.12.67
--webserver_interface=127.21.12.67
--webserver_port=0
--tserver_master_addrs=127.21.12.126:40981
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab
--principal=kudu/127.21.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:33163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:46:17.228966 8695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:17.229212 8695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:17.229285 8695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:17.233474 8695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:17.233533 8695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:17.233639 8695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.67
I20260926 16:46:17.238991 8695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:33163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.67.keytab
--principal=kudu/127.21.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.67
--webserver_port=0
--tserver_master_addrs=127.21.12.126:40981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.67
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:46:17.239992 8695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:17.240983 8695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:17.250607 8700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:17.250636 8703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:17.250754 8695 server_base.cc:1034] running on GCE node
W20260926 16:46:17.250628 8701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:17.251293 8695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:17.251940 8695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:17.253129 8695 hybrid_clock.cc:648] HybridClock initialized: now 1790441177253115 us; error 37 us; skew 500 ppm
Sep 26 16:46:17 dist-test-slave-n1hw krb5kdc[8317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441177, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:46:17.255915 8695 init.cc:377] Logged in from keytab as kudu/127.21.12.67@KRBTEST.COM (short username kudu)
I20260926 16:46:17.256748 8695 webserver.cc:460] Webserver started at http://127.21.12.67:42135/ using document root <none> and password file <none>
I20260926 16:46:17.257306 8695 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:17.257391 8695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:17.257617 8695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:46:17.259714 8695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-2/data/instance:
uuid: "d0b5245033b441a7b337358b282420d5"
format_stamp: "Formatted at 2026-09-26 16:46:17 on dist-test-slave-n1hw"
server_key: "183f0fe47915a01e950c583be89746af"
server_key_iv: "08f41cd08c541c0283048e4c36936056"
server_key_version: "encryptionkey@0"
I20260926 16:46:17.260278 8695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance:
uuid: "d0b5245033b441a7b337358b282420d5"
format_stamp: "Formatted at 2026-09-26 16:46:17 on dist-test-slave-n1hw"
server_key: "183f0fe47915a01e950c583be89746af"
server_key_iv: "08f41cd08c541c0283048e4c36936056"
server_key_version: "encryptionkey@0"
I20260926 16:46:17.264019 8695 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 16:46:17.266409 8710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:17.267724 8695 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:46:17.267863 8695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-2/wal
uuid: "d0b5245033b441a7b337358b282420d5"
format_stamp: "Formatted at 2026-09-26 16:46:17 on dist-test-slave-n1hw"
server_key: "183f0fe47915a01e950c583be89746af"
server_key_iv: "08f41cd08c541c0283048e4c36936056"
server_key_version: "encryptionkey@0"
I20260926 16:46:17.267968 8695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:17.283190 8695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:17.286700 8695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:17.286916 8695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:17.287490 8695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:46:17.288287 8695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:46:17.288353 8695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:17.288417 8695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:46:17.288463 8695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:17.296687 8695 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.67:42181
I20260926 16:46:17.296733 8823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.67:42181 every 8 connection(s)
I20260926 16:46:17.297573 8695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
Sep 26 16:46:17 dist-test-slave-n1hw krb5kdc[8317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441177, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.67@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:46:17.307788 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 8695
I20260926 16:46:17.307924 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance
I20260926 16:46:17.308285 21553 external_mini_cluster.cc:1428] Setting key 321525ce533f8a34bf267211c2bd6c85
I20260926 16:46:17.311508 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:17.299332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:58881 (local address 127.21.12.126:40981)
0926 16:46:17.299453 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:17.299457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:17.300258 (+ 801us) server_negotiation.cc:407] Connection header received
0926 16:46:17.301629 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:17.301633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:17.301692 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:17.301811 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:17.302921 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:17.303443 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:17.304790 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:17.305014 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:17.305125 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:17.307605 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:17.307608 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:17.307610 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:17.307635 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:17.309228 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:17.309886 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:17.309892 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:17.309897 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:17.309988 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:17.310335 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:17.310339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:17.310341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:17.310606 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:17.310704 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:17.311208 (+ 504us) server_negotiation.cc:299] Negotiation successful
0926 16:46:17.311325 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 16:46:17.312255 8826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:17.299707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:40981 (local address 127.21.12.67:58881)
0926 16:46:17.300126 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:17.300166 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:17.301420 (+ 1254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:17.301982 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:17.301989 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:17.302332 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:46:17.302782 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:17.302794 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:17.303686 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:17.303692 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:17.304658 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:17.304668 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:17.305176 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:17.305180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:17.305335 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:17.305968 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:46:17.305984 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:46:17.307418 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:46:17.309338 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:17.309343 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:17.309354 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:17.309728 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:17.310093 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:17.310097 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:17.310099 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:17.310241 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:17.310703 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:46:17.310711 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:46:17.310969 (+ 258us) client_negotiation.cc:769] Sending connection context
0926 16:46:17.311168 (+ 199us) client_negotiation.cc:240] Negotiation successful
0926 16:46:17.311483 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":107,"threads_started":1}
I20260926 16:46:17.313290 8824 heartbeater.cc:347] Connected to a master server at 127.21.12.126:40981
I20260926 16:46:17.313556 8824 heartbeater.cc:464] Registering TS with master...
I20260926 16:46:17.314069 8824 heartbeater.cc:511] Master 127.21.12.126:40981 requested a full tablet report, sending...
I20260926 16:46:17.315047 8363 ts_manager.cc:194] Registered new tserver with Master: d0b5245033b441a7b337358b282420d5 (127.21.12.67:42181)
I20260926 16:46:17.315615 8363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.67@KRBTEST.COM'} at 127.21.12.67:58881
I20260926 16:46:17.322173 21553 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:46:17.331089 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:17.324134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43222 (local address 127.21.12.126:40981)
0926 16:46:17.324245 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:17.324254 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:17.324372 (+ 118us) server_negotiation.cc:407] Connection header received
0926 16:46:17.324466 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:17.324468 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:17.324504 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:17.324567 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:17.325218 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:17.325735 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:17.326567 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:17.326796 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:17.326873 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:17.328181 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:17.328184 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:17.328186 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:17.328208 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:17.329622 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:17.330078 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:17.330082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:17.330083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:17.330154 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:17.330375 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:17.330377 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:17.330378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:17.330568 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:17.330702 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:17.330891 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 16:46:17.330977 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 16:46:17.334041 8363 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43222:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:46:17.337157 8363 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:46:17.351307 8836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:17.345299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34978 (local address 127.21.12.67:42181)
0926 16:46:17.346204 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:17.346217 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:17.346241 (+ 24us) server_negotiation.cc:407] Connection header received
0926 16:46:17.346422 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:17.346430 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:17.346649 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:17.346798 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:46:17.347135 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:17.348290 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:17.349843 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:17.350602 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:17.350717 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:17.350857 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:17.351071 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 16:46:17.351159 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":806,"thread_start_us":87,"threads_started":1}
I20260926 16:46:17.351320 8835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:17.345182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.67:42181 (local address 127.0.0.1:34978)
0926 16:46:17.345991 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:17.346080 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:17.346217 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:17.346791 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:17.346799 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:17.346826 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:46:17.346971 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:17.346981 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:17.348444 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:17.348447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:17.349705 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:17.349716 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:17.350730 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:17.350734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:17.350860 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:17.350903 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 16:46:17.351013 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 16:46:17.351148 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"mutex_wait_us":2,"thread_start_us":81,"threads_started":1}
I20260926 16:46:17.351586 8840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:17.346390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58244 (local address 127.21.12.65:43795)
0926 16:46:17.346791 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:17.346805 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:17.347050 (+ 245us) server_negotiation.cc:407] Connection header received
0926 16:46:17.347233 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:17.347239 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:17.347474 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:17.347635 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:46:17.347820 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:17.348872 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:17.350185 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:17.350779 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:17.350878 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:17.351013 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:17.351320 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 16:46:17.351427 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":91,"threads_started":1}
I20260926 16:46:17.351933 8839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:17.345977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:43795 (local address 127.0.0.1:58244)
0926 16:46:17.346918 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:17.346932 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:17.347079 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:17.347624 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:17.347628 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:17.347646 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:46:17.347726 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:17.347732 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:17.349017 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:17.349020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:17.350079 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:17.350087 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:17.351072 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:17.351087 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:17.351175 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:17.351196 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 16:46:17.351271 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 16:46:17.351333 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":887,"mutex_wait_us":2,"thread_start_us":106,"threads_started":1}
I20260926 16:46:17.352100 8837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:17.345359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:43311 (local address 127.0.0.1:41058)
0926 16:46:17.346373 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:17.346385 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:17.346469 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:17.347122 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:17.347125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:17.347147 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:46:17.347245 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:17.347258 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:17.348363 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:17.348366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:17.349597 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:17.349607 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:17.351640 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:17.351644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:17.351814 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:17.351837 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 16:46:17.351908 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 16:46:17.351981 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":953,"mutex_wait_us":95,"thread_start_us":93,"threads_started":1}
I20260926 16:46:17.353018 8838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:17.345534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41058 (local address 127.21.12.66:43311)
0926 16:46:17.346092 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:17.346105 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:17.346600 (+ 495us) server_negotiation.cc:407] Connection header received
0926 16:46:17.346678 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:17.346682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:17.346856 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:17.347091 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:46:17.347359 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:17.348232 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:17.349714 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:17.350602 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:17.350717 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:17.350857 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:17.352190 (+ 1333us) server_negotiation.cc:299] Negotiation successful
0926 16:46:17.352904 (+ 714us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":137,"threads_started":1}
I20260926 16:46:17.354676 8489 tablet_service.cc:1467] Processing CreateTablet for tablet 59137a1a9f2e45c7a2542c4419fbb8e8 (DEFAULT_TABLE table=test-table [id=504af96f34d84f7bbb142bcd275f75ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:46:17.354537 8758 tablet_service.cc:1467] Processing CreateTablet for tablet 59137a1a9f2e45c7a2542c4419fbb8e8 (DEFAULT_TABLE table=test-table [id=504af96f34d84f7bbb142bcd275f75ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:46:17.355198 8624 tablet_service.cc:1467] Processing CreateTablet for tablet 59137a1a9f2e45c7a2542c4419fbb8e8 (DEFAULT_TABLE table=test-table [id=504af96f34d84f7bbb142bcd275f75ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:46:17.355666 8489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59137a1a9f2e45c7a2542c4419fbb8e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:17.355672 8758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59137a1a9f2e45c7a2542c4419fbb8e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:17.356177 8624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59137a1a9f2e45c7a2542c4419fbb8e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:17.361996 8841 tablet_bootstrap.cc:492] T 59137a1a9f2e45c7a2542c4419fbb8e8 P 432c67dd5fe34190b781f112b3985bab: Bootstrap starting.
I20260926 16:46:17.363512 8843 tablet_bootstrap.cc:492] T 59137a1a9f2e45c7a2542c4419fbb8e8 P d0b5245033b441a7b337358b282420d5: Bootstrap starting.
I20260926 16:46:17.364146 8841 tablet_bootstrap.cc:654] T 59137a1a9f2e45c7a2542c4419fbb8e8 P 432c67dd5fe34190b781f112b3985bab: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:17.365159 8843 tablet_bootstrap.cc:654] T 59137a1a9f2e45c7a2542c4419fbb8e8 P d0b5245033b441a7b337358b282420d5: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:17.365195 8841 log.cc:824] T 59137a1a9f2e45c7a2542c4419fbb8e8 P 432c67dd5fe34190b781f112b3985bab: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:17.365882 8842 tablet_bootstrap.cc:492] T 59137a1a9f2e45c7a2542c4419fbb8e8 P 398bb08e3d5b43f28620ac1d5577da35: Bootstrap starting.
I20260926 16:46:17.366416 8843 log.cc:824] T 59137a1a9f2e45c7a2542c4419fbb8e8 P d0b5245033b441a7b337358b282420d5: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:17.367300 8841 tablet_bootstrap.cc:492] T 59137a1a9f2e45c7a2542c4419fbb8e8 P 432c67dd5fe34190b781f112b3985bab: No bootstrap required, opened a new log
I20260926 16:46:17.367498 8841 ts_tablet_manager.cc:1397] T 59137a1a9f2e45c7a2542c4419fbb8e8 P 432c67dd5fe34190b781f112b3985bab: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 16:46:17.367916 8842 tablet_bootstrap.cc:654] T 59137a1a9f2e45c7a2542c4419fbb8e8 P 398bb08e3d5b43f28620ac1d5577da35: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:17.368395 8843 tablet_bootstrap.cc:492] T 59137a1a9f2e45c7a2542c4419fbb8e8 P d0b5245033b441a7b337358b282420d5: No bootstrap required, opened a new log
I20260926 16:46:17.368598 8843 ts_tablet_manager.cc:1397] T 59137a1a9f2e45c7a2542c4419fbb8e8 P d0b5245033b441a7b337358b282420d5: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 16:46:17.369097 8842 log.cc:824] T 59137a1a9f2e45c7a2542c4419fbb8e8 P 398bb08e3d5b43f28620ac1d5577da35: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:17.370687 8841 raft_consensus.cc:348] T 59137a1a9f2e45c7a2542c4419fbb8e8 P 432c67dd5fe34190b781f112b3985bab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b5245033b441a7b337358b282420d5" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42181 } } peers { permanent_uuid: "432c67dd5fe34190b781f112b3985bab" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43311 } } peers { permanent_uuid: "398bb08e3d5b43f28620ac1d5577da35" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 43795 } }
I20260926 16:46:17.370926 8841 raft_consensus.cc:374] T 59137a1a9f2e45c7a2542c4419fbb8e8 P 432c67dd5fe34190b781f112b3985bab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:46:17.370985 8841 raft_consensus.cc:729] T 59137a1a9f2e45c7a2542c4419fbb8e8 P 432c67dd5fe34190b781f112b3985bab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 432c67dd5fe34190b781f112b3985bab, State: Initialized, Role: FOLLOWER
I20260926 16:46:17.371001 8842 tablet_bootstrap.cc:492] T 59137a1a9f2e45c7a2542c4419fbb8e8 P 398bb08e3d5b43f28620ac1d5577da35: No bootstrap required, opened a new log
I20260926 16:46:17.371181 8842 ts_tablet_manager.cc:1397] T 59137a1a9f2e45c7a2542c4419fbb8e8 P 398bb08e3d5b43f28620ac1d5577da35: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 16:46:17.371507 8841 consensus_queue.cc:260] T 59137a1a9f2e45c7a2542c4419fbb8e8 P 432c67dd5fe34190b781f112b3985bab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b5245033b441a7b337358b282420d5" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42181 } } peers { permanent_uuid: "432c67dd5fe34190b781f112b3985bab" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43311 } } peers { permanent_uuid: "398bb08e3d5b43f28620ac1d5577da35" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 43795 } }
I20260926 16:46:17.372143 8690 heartbeater.cc:503] Master 127.21.12.126:40981 was elected leader, sending a full tablet report...
I20260926 16:46:17.372529 8841 ts_tablet_manager.cc:1428] T 59137a1a9f2e45c7a2542c4419fbb8e8 P 432c67dd5fe34190b781f112b3985bab: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 16:46:17.372327 8843 raft_consensus.cc:348] T 59137a1a9f2e45c7a2542c4419fbb8e8 P d0b5245033b441a7b337358b282420d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b5245033b441a7b337358b282420d5" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42181 } } peers { permanent_uuid: "432c67dd5fe34190b781f112b3985bab" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43311 } } peers { permanent_uuid: "398bb08e3d5b43f28620ac1d5577da35" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 43795 } }
I20260926 16:46:17.372622 8843 raft_consensus.cc:374] T 59137a1a9f2e45c7a2542c4419fbb8e8 P d0b5245033b441a7b337358b282420d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:46:17.372669 8843 raft_consensus.cc:729] T 59137a1a9f2e45c7a2542c4419fbb8e8 P d0b5245033b441a7b337358b282420d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0b5245033b441a7b337358b282420d5, State: Initialized, Role: FOLLOWER
I20260926 16:46:17.373231 8843 consensus_queue.cc:260] T 59137a1a9f2e45c7a2542c4419fbb8e8 P d0b5245033b441a7b337358b282420d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b5245033b441a7b337358b282420d5" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42181 } } peers { permanent_uuid: "432c67dd5fe34190b781f112b3985bab" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43311 } } peers { permanent_uuid: "398bb08e3d5b43f28620ac1d5577da35" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 43795 } }
I20260926 16:46:17.374156 8843 ts_tablet_manager.cc:1428] T 59137a1a9f2e45c7a2542c4419fbb8e8 P d0b5245033b441a7b337358b282420d5: Time spent starting tablet: real 0.005s user 0.002s sys 0.005s
I20260926 16:46:17.374490 8824 heartbeater.cc:503] Master 127.21.12.126:40981 was elected leader, sending a full tablet report...
I20260926 16:46:17.375166 8842 raft_consensus.cc:348] T 59137a1a9f2e45c7a2542c4419fbb8e8 P 398bb08e3d5b43f28620ac1d5577da35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b5245033b441a7b337358b282420d5" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42181 } } peers { permanent_uuid: "432c67dd5fe34190b781f112b3985bab" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43311 } } peers { permanent_uuid: "398bb08e3d5b43f28620ac1d5577da35" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 43795 } }
I20260926 16:46:17.375387 8842 raft_consensus.cc:374] T 59137a1a9f2e45c7a2542c4419fbb8e8 P 398bb08e3d5b43f28620ac1d5577da35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:46:17.375461 8842 raft_consensus.cc:729] T 59137a1a9f2e45c7a2542c4419fbb8e8 P 398bb08e3d5b43f28620ac1d5577da35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398bb08e3d5b43f28620ac1d5577da35, State: Initialized, Role: FOLLOWER
I20260926 16:46:17.375877 8842 consensus_queue.cc:260] T 59137a1a9f2e45c7a2542c4419fbb8e8 P 398bb08e3d5b43f28620ac1d5577da35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b5245033b441a7b337358b282420d5" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42181 } } peers { permanent_uuid: "432c67dd5fe34190b781f112b3985bab" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43311 } } peers { permanent_uuid: "398bb08e3d5b43f28620ac1d5577da35" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 43795 } }
I20260926 16:46:17.376662 8555 heartbeater.cc:503] Master 127.21.12.126:40981 was elected leader, sending a full tablet report...
I20260926 16:46:17.376775 8842 ts_tablet_manager.cc:1428] T 59137a1a9f2e45c7a2542c4419fbb8e8 P 398bb08e3d5b43f28620ac1d5577da35: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 16:46:17.386649 8847 raft_consensus.cc:482] T 59137a1a9f2e45c7a2542c4419fbb8e8 P 432c67dd5fe34190b781f112b3985bab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:46:17.386811 8847 raft_consensus.cc:504] T 59137a1a9f2e45c7a2542c4419fbb8e8 P 432c67dd5fe34190b781f112b3985bab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b5245033b441a7b337358b282420d5" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42181 } } peers { permanent_uuid: "432c67dd5fe34190b781f112b3985bab" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43311 } } peers { permanent_uuid: "398bb08e3d5b43f28620ac1d5577da35" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 43795 } }
I20260926 16:46:17.387787 8847 leader_election.cc:288] T 59137a1a9f2e45c7a2542c4419fbb8e8 P 432c67dd5fe34190b781f112b3985bab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0b5245033b441a7b337358b282420d5 (127.21.12.67:42181), 398bb08e3d5b43f28620ac1d5577da35 (127.21.12.65:43795)
I20260926 16:46:17.391446 8692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:17.388046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:43795 (local address 127.21.12.66:48271)
0926 16:46:17.388217 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:17.388233 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:17.388330 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:17.388637 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:17.388640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:17.388658 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:46:17.388795 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:17.388802 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:17.389562 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:17.389568 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:17.390434 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:17.390443 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:17.391039 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:17.391043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:17.391161 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:17.391186 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 16:46:17.391253 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 16:46:17.391338 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260926 16:46:17.391532 8840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:17.388082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:48271 (local address 127.21.12.65:43795)
0926 16:46:17.388213 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:17.388218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:17.388288 (+ 70us) server_negotiation.cc:407] Connection header received
0926 16:46:17.388445 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:17.388449 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:17.388509 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:17.388586 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:46:17.388905 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:17.389451 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:17.390555 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:17.390927 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:17.391065 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:17.391121 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:17.391298 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 16:46:17.391376 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 16:46:17.391681 8850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:17.388066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.67:42181 (local address 127.21.12.66:40907)
0926 16:46:17.388431 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:17.388445 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:17.388549 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:17.388783 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:17.388785 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:17.388799 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:46:17.388916 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:17.388923 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:17.389620 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:17.389622 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:17.390437 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:17.390445 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:17.391258 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:17.391283 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:17.391384 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:17.391407 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 16:46:17.391475 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 16:46:17.391545 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":97,"threads_started":1}
I20260926 16:46:17.391752 8836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:17.388203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:40907 (local address 127.21.12.67:42181)
0926 16:46:17.388311 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:17.388315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:17.388519 (+ 204us) server_negotiation.cc:407] Connection header received
0926 16:46:17.388647 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:17.388650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:17.388692 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:17.388769 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:46:17.389017 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:17.389479 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:17.390562 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:17.390950 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:17.391085 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:17.391147 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:17.391523 (+ 376us) server_negotiation.cc:299] Negotiation successful
0926 16:46:17.391605 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 16:46:17.392051 8509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59137a1a9f2e45c7a2542c4419fbb8e8" candidate_uuid: "432c67dd5fe34190b781f112b3985bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398bb08e3d5b43f28620ac1d5577da35" is_pre_election: true
I20260926 16:46:17.392278 8778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59137a1a9f2e45c7a2542c4419fbb8e8" candidate_uuid: "432c67dd5fe34190b781f112b3985bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0b5245033b441a7b337358b282420d5" is_pre_election: true
I20260926 16:46:17.392333 8509 raft_consensus.cc:2454] T 59137a1a9f2e45c7a2542c4419fbb8e8 P 398bb08e3d5b43f28620ac1d5577da35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 432c67dd5fe34190b781f112b3985bab in term 0.
I20260926 16:46:17.392530 8778 raft_consensus.cc:2454] T 59137a1a9f2e45c7a2542c4419fbb8e8 P d0b5245033b441a7b337358b282420d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 432c67dd5fe34190b781f112b3985bab in term 0.
I20260926 16:46:17.392851 8576 leader_election.cc:302] T 59137a1a9f2e45c7a2542c4419fbb8e8 P 432c67dd5fe34190b781f112b3985bab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398bb08e3d5b43f28620ac1d5577da35, 432c67dd5fe34190b781f112b3985bab; no voters:
I20260926 16:46:17.393116 8847 raft_consensus.cc:2781] T 59137a1a9f2e45c7a2542c4419fbb8e8 P 432c67dd5fe34190b781f112b3985bab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:46:17.393208 8847 raft_consensus.cc:482] T 59137a1a9f2e45c7a2542c4419fbb8e8 P 432c67dd5fe34190b781f112b3985bab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:46:17.393266 8847 raft_consensus.cc:3037] T 59137a1a9f2e45c7a2542c4419fbb8e8 P 432c67dd5fe34190b781f112b3985bab [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:17.394275 8847 raft_consensus.cc:504] T 59137a1a9f2e45c7a2542c4419fbb8e8 P 432c67dd5fe34190b781f112b3985bab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b5245033b441a7b337358b282420d5" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42181 } } peers { permanent_uuid: "432c67dd5fe34190b781f112b3985bab" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43311 } } peers { permanent_uuid: "398bb08e3d5b43f28620ac1d5577da35" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 43795 } }
I20260926 16:46:17.394608 8847 leader_election.cc:288] T 59137a1a9f2e45c7a2542c4419fbb8e8 P 432c67dd5fe34190b781f112b3985bab [CANDIDATE]: Term 1 election: Requested vote from peers d0b5245033b441a7b337358b282420d5 (127.21.12.67:42181), 398bb08e3d5b43f28620ac1d5577da35 (127.21.12.65:43795)
I20260926 16:46:17.394927 8778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59137a1a9f2e45c7a2542c4419fbb8e8" candidate_uuid: "432c67dd5fe34190b781f112b3985bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0b5245033b441a7b337358b282420d5"
I20260926 16:46:17.394927 8509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59137a1a9f2e45c7a2542c4419fbb8e8" candidate_uuid: "432c67dd5fe34190b781f112b3985bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398bb08e3d5b43f28620ac1d5577da35"
I20260926 16:46:17.395040 8509 raft_consensus.cc:3037] T 59137a1a9f2e45c7a2542c4419fbb8e8 P 398bb08e3d5b43f28620ac1d5577da35 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:17.395048 8778 raft_consensus.cc:3037] T 59137a1a9f2e45c7a2542c4419fbb8e8 P d0b5245033b441a7b337358b282420d5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:17.396279 8778 raft_consensus.cc:2454] T 59137a1a9f2e45c7a2542c4419fbb8e8 P d0b5245033b441a7b337358b282420d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 432c67dd5fe34190b781f112b3985bab in term 1.
I20260926 16:46:17.396279 8509 raft_consensus.cc:2454] T 59137a1a9f2e45c7a2542c4419fbb8e8 P 398bb08e3d5b43f28620ac1d5577da35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 432c67dd5fe34190b781f112b3985bab in term 1.
I20260926 16:46:17.396598 8578 leader_election.cc:302] T 59137a1a9f2e45c7a2542c4419fbb8e8 P 432c67dd5fe34190b781f112b3985bab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 398bb08e3d5b43f28620ac1d5577da35, 432c67dd5fe34190b781f112b3985bab, d0b5245033b441a7b337358b282420d5; no voters:
I20260926 16:46:17.396800 8847 raft_consensus.cc:2781] T 59137a1a9f2e45c7a2542c4419fbb8e8 P 432c67dd5fe34190b781f112b3985bab [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:46:17.397081 8847 raft_consensus.cc:686] T 59137a1a9f2e45c7a2542c4419fbb8e8 P 432c67dd5fe34190b781f112b3985bab [term 1 LEADER]: Becoming Leader. State: Replica: 432c67dd5fe34190b781f112b3985bab, State: Running, Role: LEADER
I20260926 16:46:17.397475 8847 consensus_queue.cc:237] T 59137a1a9f2e45c7a2542c4419fbb8e8 P 432c67dd5fe34190b781f112b3985bab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b5245033b441a7b337358b282420d5" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42181 } } peers { permanent_uuid: "432c67dd5fe34190b781f112b3985bab" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43311 } } peers { permanent_uuid: "398bb08e3d5b43f28620ac1d5577da35" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 43795 } }
I20260926 16:46:17.400488 8362 catalog_manager.cc:5495] T 59137a1a9f2e45c7a2542c4419fbb8e8 P 432c67dd5fe34190b781f112b3985bab reported cstate change: term changed from 0 to 1, leader changed from <none> to 432c67dd5fe34190b781f112b3985bab (127.21.12.66). New cstate: current_term: 1 leader_uuid: "432c67dd5fe34190b781f112b3985bab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b5245033b441a7b337358b282420d5" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "432c67dd5fe34190b781f112b3985bab" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "398bb08e3d5b43f28620ac1d5577da35" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 43795 } health_report { overall_health: UNKNOWN } } }
I20260926 16:46:17.413012 8838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:17.409514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41070 (local address 127.21.12.66:43311)
0926 16:46:17.409625 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:17.409629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:17.409641 (+ 12us) server_negotiation.cc:407] Connection header received
0926 16:46:17.409746 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:17.409791 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:17.409852 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:17.409961 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:46:17.410228 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:17.410754 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:17.411611 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:17.411829 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:17.411932 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:17.412236 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:46:17.412622 (+ 386us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:46:17.412748 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:17.412807 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 16:46:17.412858 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 16:46:17.418478 8778 raft_consensus.cc:1261] T 59137a1a9f2e45c7a2542c4419fbb8e8 P d0b5245033b441a7b337358b282420d5 [term 1 FOLLOWER]: Refusing update from remote peer 432c67dd5fe34190b781f112b3985bab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:46:17.418499 8509 raft_consensus.cc:1261] T 59137a1a9f2e45c7a2542c4419fbb8e8 P 398bb08e3d5b43f28620ac1d5577da35 [term 1 FOLLOWER]: Refusing update from remote peer 432c67dd5fe34190b781f112b3985bab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:46:17.419138 8851 consensus_queue.cc:1035] T 59137a1a9f2e45c7a2542c4419fbb8e8 P 432c67dd5fe34190b781f112b3985bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0b5245033b441a7b337358b282420d5" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 42181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:46:17.419309 8847 consensus_queue.cc:1035] T 59137a1a9f2e45c7a2542c4419fbb8e8 P 432c67dd5fe34190b781f112b3985bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "398bb08e3d5b43f28620ac1d5577da35" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 43795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:46:17.428758 8855 mvcc.cc:205] Tried to move back new op lower bound from 7333647062699184128 to 7333647062622793728. Current Snapshot: MvccSnapshot[applied={T|T < 7333647062699184128}]
I20260926 16:46:17.434650 8362 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:46:17.434845 8362 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:46:17.437299 8362 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bed454a2f8964c8692d424b730206f54: Sending DeleteTablet for 3 replicas of tablet 59137a1a9f2e45c7a2542c4419fbb8e8
I20260926 16:46:17.438170 8624 tablet_service.cc:1514] Processing DeleteTablet for tablet 59137a1a9f2e45c7a2542c4419fbb8e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:46:17 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:41058
I20260926 16:46:17.438169 8758 tablet_service.cc:1514] Processing DeleteTablet for tablet 59137a1a9f2e45c7a2542c4419fbb8e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:46:17 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:34978
I20260926 16:46:17.438699 8863 tablet_replica.cc:330] T 59137a1a9f2e45c7a2542c4419fbb8e8 P 432c67dd5fe34190b781f112b3985bab: stopping tablet replica
I20260926 16:46:17.438700 8489 tablet_service.cc:1514] Processing DeleteTablet for tablet 59137a1a9f2e45c7a2542c4419fbb8e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:46:17 UTC) from {username='kudu', principal='kudu/127.21.12.126@KRBTEST.COM'} at 127.0.0.1:58244
I20260926 16:46:17.439040 8863 raft_consensus.cc:2229] T 59137a1a9f2e45c7a2542c4419fbb8e8 P 432c67dd5fe34190b781f112b3985bab [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:46:17.439203 8865 tablet_replica.cc:330] T 59137a1a9f2e45c7a2542c4419fbb8e8 P 398bb08e3d5b43f28620ac1d5577da35: stopping tablet replica
I20260926 16:46:17.439440 8863 raft_consensus.cc:2258] T 59137a1a9f2e45c7a2542c4419fbb8e8 P 432c67dd5fe34190b781f112b3985bab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:46:17.439604 8865 raft_consensus.cc:2229] T 59137a1a9f2e45c7a2542c4419fbb8e8 P 398bb08e3d5b43f28620ac1d5577da35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:46:17.439911 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 8425
I20260926 16:46:17.441716 8863 ts_tablet_manager.cc:1905] T 59137a1a9f2e45c7a2542c4419fbb8e8 P 432c67dd5fe34190b781f112b3985bab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:46:17.444490 8863 ts_tablet_manager.cc:1918] T 59137a1a9f2e45c7a2542c4419fbb8e8 P 432c67dd5fe34190b781f112b3985bab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:46:17.444610 8863 log.cc:1196] T 59137a1a9f2e45c7a2542c4419fbb8e8 P 432c67dd5fe34190b781f112b3985bab: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-1/wal/wals/59137a1a9f2e45c7a2542c4419fbb8e8
I20260926 16:46:17.444900 8863 ts_tablet_manager.cc:1939] T 59137a1a9f2e45c7a2542c4419fbb8e8 P 432c67dd5fe34190b781f112b3985bab: Deleting consensus metadata
I20260926 16:46:17.446050 8350 catalog_manager.cc:4841] TS 432c67dd5fe34190b781f112b3985bab (127.21.12.66:43311): tablet 59137a1a9f2e45c7a2542c4419fbb8e8 (table test-table [id=504af96f34d84f7bbb142bcd275f75ae]) successfully deleted
W20260926 16:46:17.448485 8347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.12.65:43795 (error 108)
I20260926 16:46:17.448580 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 8560
I20260926 16:46:17.448662 8864 tablet_replica.cc:330] T 59137a1a9f2e45c7a2542c4419fbb8e8 P d0b5245033b441a7b337358b282420d5: stopping tablet replica
I20260926 16:46:17.448952 8864 raft_consensus.cc:2229] T 59137a1a9f2e45c7a2542c4419fbb8e8 P d0b5245033b441a7b337358b282420d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:46:17.449263 8864 raft_consensus.cc:2258] T 59137a1a9f2e45c7a2542c4419fbb8e8 P d0b5245033b441a7b337358b282420d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:46:17.450141 8835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:17.449894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:43795 (local address 127.0.0.1:58260)
0926 16:46:17.450002 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:17.450074 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.65:43795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260926 16:46:17.450275 8347 catalog_manager.cc:4584] TS 398bb08e3d5b43f28620ac1d5577da35 (127.21.12.65:43795): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 59137a1a9f2e45c7a2542c4419fbb8e8: Network error: Client connection negotiation failed: client connection to 127.21.12.65:43795: connect: Connection refused (error 111)
I20260926 16:46:17.451503 8864 ts_tablet_manager.cc:1905] T 59137a1a9f2e45c7a2542c4419fbb8e8 P d0b5245033b441a7b337358b282420d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:46:17.453660 8864 ts_tablet_manager.cc:1918] T 59137a1a9f2e45c7a2542c4419fbb8e8 P d0b5245033b441a7b337358b282420d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:46:17.453738 8864 log.cc:1196] T 59137a1a9f2e45c7a2542c4419fbb8e8 P d0b5245033b441a7b337358b282420d5: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440924937351-21553-0/minicluster-data/ts-2/wal/wals/59137a1a9f2e45c7a2542c4419fbb8e8
I20260926 16:46:17.454078 8864 ts_tablet_manager.cc:1939] T 59137a1a9f2e45c7a2542c4419fbb8e8 P d0b5245033b441a7b337358b282420d5: Deleting consensus metadata
I20260926 16:46:17.455139 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 8695
I20260926 16:46:17.455205 8349 catalog_manager.cc:4841] TS d0b5245033b441a7b337358b282420d5 (127.21.12.67:42181): tablet 59137a1a9f2e45c7a2542c4419fbb8e8 (table test-table [id=504af96f34d84f7bbb142bcd275f75ae]) successfully deleted
I20260926 16:46:17.460515 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 8330
2026-09-26T16:46:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2239 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T16:46:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:46:17Z Disabled control of system clock
I20260926 16:46:17.490572 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:34849
--webserver_interface=127.21.12.126
--webserver_port=0
--builtin_ntp_servers=127.21.12.84:33761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:34849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 16:46:17.596226 8870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:17.596482 8870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:17.596516 8870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:17.600420 8870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:46:17.600473 8870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:17.600489 8870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:17.600505 8870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:46:17.600520 8870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:46:17.605113 8870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:33761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:34849
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:34849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:46:17.606135 8870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:17.607399 8870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:17.615504 8878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:17.615504 8876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:17.615547 8875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:17.615655 8870 server_base.cc:1034] running on GCE node
I20260926 16:46:17.616328 8870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:17.617383 8870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:17.618559 8870 hybrid_clock.cc:648] HybridClock initialized: now 1790441177618536 us; error 37 us; skew 500 ppm
I20260926 16:46:17.620393 8870 webserver.cc:460] Webserver started at http://127.21.12.126:38555/ using document root <none> and password file <none>
I20260926 16:46:17.621096 8870 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:17.621196 8870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:17.621436 8870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:46:17.623736 8870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "f9e6b25430044db382aae127ad7a72a9"
format_stamp: "Formatted at 2026-09-26 16:46:17 on dist-test-slave-n1hw"
server_key: "4a2963266cebdf1c2ed67eb968fcc6ff"
server_key_iv: "13a0f7e5c566c4b12bb3e656883d3327"
server_key_version: "encryptionkey@0"
I20260926 16:46:17.624336 8870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "f9e6b25430044db382aae127ad7a72a9"
format_stamp: "Formatted at 2026-09-26 16:46:17 on dist-test-slave-n1hw"
server_key: "4a2963266cebdf1c2ed67eb968fcc6ff"
server_key_iv: "13a0f7e5c566c4b12bb3e656883d3327"
server_key_version: "encryptionkey@0"
I20260926 16:46:17.628163 8870 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 16:46:17.630555 8884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:17.631774 8870 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:46:17.631906 8870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "f9e6b25430044db382aae127ad7a72a9"
format_stamp: "Formatted at 2026-09-26 16:46:17 on dist-test-slave-n1hw"
server_key: "4a2963266cebdf1c2ed67eb968fcc6ff"
server_key_iv: "13a0f7e5c566c4b12bb3e656883d3327"
server_key_version: "encryptionkey@0"
I20260926 16:46:17.632009 8870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:17.654300 8870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:17.655041 8870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:17.655242 8870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:17.662402 8870 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.126:34849
I20260926 16:46:17.662417 8936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.126:34849 every 8 connection(s)
I20260926 16:46:17.663333 8870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:46:17.665884 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 8870
I20260926 16:46:17.666025 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:46:17.666333 21553 external_mini_cluster.cc:1428] Setting key 6003490c46c1f53604fc549342d6ecd5
I20260926 16:46:17.666756 8937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:17.673511 8937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9e6b25430044db382aae127ad7a72a9: Bootstrap starting.
I20260926 16:46:17.675851 8937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9e6b25430044db382aae127ad7a72a9: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:17.675913 8940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:17.667934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46144 (local address 127.21.12.126:34849)
0926 16:46:17.668606 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:17.668622 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:17.668656 (+ 34us) server_negotiation.cc:407] Connection header received
0926 16:46:17.669344 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:17.669362 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:17.669860 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:17.670151 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:17.670494 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:17.671462 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:17.672677 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:17.673123 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:17.673232 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:17.674181 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:17.674186 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:17.674200 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:46:17.674604 (+ 404us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:17.674663 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:46:17.674682 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:17.674757 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:17.674911 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 16:46:17.675196 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":140,"threads_started":1}
I20260926 16:46:17.676746 8937 log.cc:824] T 00000000000000000000000000000000 P f9e6b25430044db382aae127ad7a72a9: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:17.678689 8937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9e6b25430044db382aae127ad7a72a9: No bootstrap required, opened a new log
I20260926 16:46:17.682045 8937 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9e6b25430044db382aae127ad7a72a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9e6b25430044db382aae127ad7a72a9" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34849 } }
I20260926 16:46:17.682255 8937 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9e6b25430044db382aae127ad7a72a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:46:17.682288 8937 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9e6b25430044db382aae127ad7a72a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9e6b25430044db382aae127ad7a72a9, State: Initialized, Role: FOLLOWER
I20260926 16:46:17.682619 8937 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9e6b25430044db382aae127ad7a72a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9e6b25430044db382aae127ad7a72a9" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34849 } }
I20260926 16:46:17.682721 8937 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9e6b25430044db382aae127ad7a72a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:46:17.682758 8937 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9e6b25430044db382aae127ad7a72a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:46:17.682823 8937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9e6b25430044db382aae127ad7a72a9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:17.683571 8937 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9e6b25430044db382aae127ad7a72a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9e6b25430044db382aae127ad7a72a9" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34849 } }
I20260926 16:46:17.683848 8937 leader_election.cc:302] T 00000000000000000000000000000000 P f9e6b25430044db382aae127ad7a72a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9e6b25430044db382aae127ad7a72a9; no voters:
I20260926 16:46:17.684083 8937 leader_election.cc:288] T 00000000000000000000000000000000 P f9e6b25430044db382aae127ad7a72a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:46:17.684216 8942 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9e6b25430044db382aae127ad7a72a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:46:17.684448 8942 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9e6b25430044db382aae127ad7a72a9 [term 1 LEADER]: Becoming Leader. State: Replica: f9e6b25430044db382aae127ad7a72a9, State: Running, Role: LEADER
I20260926 16:46:17.684751 8942 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9e6b25430044db382aae127ad7a72a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9e6b25430044db382aae127ad7a72a9" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34849 } }
I20260926 16:46:17.685230 8937 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9e6b25430044db382aae127ad7a72a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:46:17.686311 8943 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9e6b25430044db382aae127ad7a72a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9e6b25430044db382aae127ad7a72a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9e6b25430044db382aae127ad7a72a9" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34849 } } }
I20260926 16:46:17.686434 8943 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9e6b25430044db382aae127ad7a72a9 [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:17.686815 8944 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9e6b25430044db382aae127ad7a72a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9e6b25430044db382aae127ad7a72a9. Latest consensus state: current_term: 1 leader_uuid: "f9e6b25430044db382aae127ad7a72a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9e6b25430044db382aae127ad7a72a9" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 34849 } } }
I20260926 16:46:17.686965 8944 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9e6b25430044db382aae127ad7a72a9 [sys.catalog]: This master's current role is: LEADER
W20260926 16:46:17.689805 8957 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f9e6b25430044db382aae127ad7a72a9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 16:46:17.689885 8957 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 16:46:17.689952 8951 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:46:17.690829 8951 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:46:17.696234 8951 catalog_manager.cc:1347] Generated new cluster ID: f46ba5d3dbbd48b98070ce4e34945848
I20260926 16:46:17.696298 8951 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:46:17.710722 8951 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:46:17.711987 8951 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:46:17.720360 8951 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9e6b25430044db382aae127ad7a72a9: Generated new TSK 0
I20260926 16:46:17.721230 8951 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:46:17.725248 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.65:0
--local_ip_for_outbound_sockets=127.21.12.65
--webserver_interface=127.21.12.65
--webserver_port=0
--tserver_master_addrs=127.21.12.126:34849
--builtin_ntp_servers=127.21.12.84:33761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 16:46:17.835831 8961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:17.836159 8961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:17.836230 8961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:17.840258 8961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:17.840337 8961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:17.840456 8961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.65
I20260926 16:46:17.845220 8961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:33761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.65
--webserver_port=0
--tserver_master_addrs=127.21.12.126:34849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.65
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:46:17.846215 8961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:17.847499 8961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:17.856784 8966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:17.856858 8961 server_base.cc:1034] running on GCE node
W20260926 16:46:17.856818 8967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:17.856784 8969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:17.857432 8961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:17.858150 8961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:17.859332 8961 hybrid_clock.cc:648] HybridClock initialized: now 1790441177859316 us; error 28 us; skew 500 ppm
I20260926 16:46:17.861176 8961 webserver.cc:460] Webserver started at http://127.21.12.65:36923/ using document root <none> and password file <none>
I20260926 16:46:17.861805 8961 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:17.861900 8961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:17.862120 8961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:46:17.864105 8961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-0/data/instance:
uuid: "917f917868ee42dd9ea6977ef896a7c9"
format_stamp: "Formatted at 2026-09-26 16:46:17 on dist-test-slave-n1hw"
server_key: "d75f7389de8c4aaaee79daab439a45ae"
server_key_iv: "ea8d8736ce65fca1bd95eb0a7acce1af"
server_key_version: "encryptionkey@0"
I20260926 16:46:17.864665 8961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance:
uuid: "917f917868ee42dd9ea6977ef896a7c9"
format_stamp: "Formatted at 2026-09-26 16:46:17 on dist-test-slave-n1hw"
server_key: "d75f7389de8c4aaaee79daab439a45ae"
server_key_iv: "ea8d8736ce65fca1bd95eb0a7acce1af"
server_key_version: "encryptionkey@0"
I20260926 16:46:17.868351 8961 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260926 16:46:17.870995 8975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:17.872272 8961 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 16:46:17.872411 8961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-0/wal
uuid: "917f917868ee42dd9ea6977ef896a7c9"
format_stamp: "Formatted at 2026-09-26 16:46:17 on dist-test-slave-n1hw"
server_key: "d75f7389de8c4aaaee79daab439a45ae"
server_key_iv: "ea8d8736ce65fca1bd95eb0a7acce1af"
server_key_version: "encryptionkey@0"
I20260926 16:46:17.872529 8961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:17.899642 8961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:17.900486 8961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:17.900708 8961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:17.901345 8961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:46:17.902356 8961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:46:17.902429 8961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:17.902503 8961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:46:17.902551 8961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:17.912230 8961 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.65:36101
I20260926 16:46:17.912268 9088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.65:36101 every 8 connection(s)
I20260926 16:46:17.913156 8961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-0/data/info.pb
I20260926 16:46:17.922183 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 8961
I20260926 16:46:17.922326 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-0/wal/instance
I20260926 16:46:17.922700 21553 external_mini_cluster.cc:1428] Setting key fd7559a3f4a66080c453f08169b06f84
I20260926 16:46:17.923144 8940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:17.914909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.65:41445 (local address 127.21.12.126:34849)
0926 16:46:17.915043 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:17.915047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:17.916094 (+ 1047us) server_negotiation.cc:407] Connection header received
0926 16:46:17.917444 (+ 1350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:17.917447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:17.917500 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:17.917587 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:17.918463 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:17.919038 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:17.920145 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:17.920376 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:17.920463 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:17.922379 (+ 1916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:17.922383 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:17.922387 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:46:17.922405 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:17.922428 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:46:17.922439 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:17.922492 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:17.922753 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 16:46:17.922839 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 16:46:17.924947 9091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:17.915327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34849 (local address 127.21.12.65:41445)
0926 16:46:17.915968 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:17.916008 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:17.917242 (+ 1234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:17.917778 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:17.917785 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:17.917873 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:46:17.918336 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:17.918347 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:17.919170 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:17.919172 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:17.920035 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:17.920044 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:17.920537 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:17.920539 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:17.920868 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:17.922004 (+ 1136us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:46:17.922006 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:46:17.922064 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:46:17.922150 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:46:17.922604 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:46:17.922613 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:46:17.922619 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 16:46:17.922780 (+ 161us) client_negotiation.cc:240] Negotiation successful
0926 16:46:17.923582 (+ 802us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":116,"threads_started":1}
I20260926 16:46:17.925316 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.66:0
--local_ip_for_outbound_sockets=127.21.12.66
--webserver_interface=127.21.12.66
--webserver_port=0
--tserver_master_addrs=127.21.12.126:34849
--builtin_ntp_servers=127.21.12.84:33761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 16:46:17.926807 9089 heartbeater.cc:347] Connected to a master server at 127.21.12.126:34849
I20260926 16:46:17.927090 9089 heartbeater.cc:464] Registering TS with master...
I20260926 16:46:17.927749 9089 heartbeater.cc:511] Master 127.21.12.126:34849 requested a full tablet report, sending...
I20260926 16:46:17.930145 8901 ts_manager.cc:194] Registered new tserver with Master: 917f917868ee42dd9ea6977ef896a7c9 (127.21.12.65:36101)
I20260926 16:46:17.932067 8901 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.12.65:41445
W20260926 16:46:18.046350 9092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:18.046643 9092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:18.046711 9092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:18.050740 9092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:18.050820 9092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:18.050906 9092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.66
I20260926 16:46:18.055948 9092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:33761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.66
--webserver_port=0
--tserver_master_addrs=127.21.12.126:34849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.66
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:46:18.057003 9092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:18.058473 9092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:18.068365 9097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:18.068420 9098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:18.068579 9092 server_base.cc:1034] running on GCE node
W20260926 16:46:18.068439 9100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:18.069068 9092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:18.069686 9092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:18.070844 9092 hybrid_clock.cc:648] HybridClock initialized: now 1790441178070807 us; error 47 us; skew 500 ppm
I20260926 16:46:18.072723 9092 webserver.cc:460] Webserver started at http://127.21.12.66:38737/ using document root <none> and password file <none>
I20260926 16:46:18.073294 9092 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:18.073378 9092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:18.073632 9092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:46:18.075779 9092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-1/data/instance:
uuid: "29af901fc52f4087a777887abb50ee0a"
format_stamp: "Formatted at 2026-09-26 16:46:18 on dist-test-slave-n1hw"
server_key: "c184c53ff2159443ed31c0c669796731"
server_key_iv: "3c3fe2f1287a599f4efb5990737572d0"
server_key_version: "encryptionkey@0"
I20260926 16:46:18.076333 9092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance:
uuid: "29af901fc52f4087a777887abb50ee0a"
format_stamp: "Formatted at 2026-09-26 16:46:18 on dist-test-slave-n1hw"
server_key: "c184c53ff2159443ed31c0c669796731"
server_key_iv: "3c3fe2f1287a599f4efb5990737572d0"
server_key_version: "encryptionkey@0"
I20260926 16:46:18.079988 9092 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 16:46:18.082458 9106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:18.083757 9092 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:46:18.083901 9092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-1/wal
uuid: "29af901fc52f4087a777887abb50ee0a"
format_stamp: "Formatted at 2026-09-26 16:46:18 on dist-test-slave-n1hw"
server_key: "c184c53ff2159443ed31c0c669796731"
server_key_iv: "3c3fe2f1287a599f4efb5990737572d0"
server_key_version: "encryptionkey@0"
I20260926 16:46:18.084015 9092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:18.095960 9092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:18.096565 9092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:18.096748 9092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:18.097283 9092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:46:18.098246 9092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:46:18.098312 9092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:18.098376 9092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:46:18.098423 9092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:18.107227 9092 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.66:38453
I20260926 16:46:18.107273 9219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.66:38453 every 8 connection(s)
I20260926 16:46:18.108122 9092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-1/data/info.pb
I20260926 16:46:18.115114 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 9092
I20260926 16:46:18.115353 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-1/wal/instance
I20260926 16:46:18.115639 21553 external_mini_cluster.cc:1428] Setting key ebaeef15d83fbe69c71beaec43534d1b
I20260926 16:46:18.117653 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.67:0
--local_ip_for_outbound_sockets=127.21.12.67
--webserver_interface=127.21.12.67
--webserver_port=0
--tserver_master_addrs=127.21.12.126:34849
--builtin_ntp_servers=127.21.12.84:33761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 16:46:18.118059 8940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:18.110042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:60661 (local address 127.21.12.126:34849)
0926 16:46:18.110182 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:18.110186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:18.111181 (+ 995us) server_negotiation.cc:407] Connection header received
0926 16:46:18.112796 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:18.112800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:18.112865 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:18.112968 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:18.113971 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:18.114546 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:18.115907 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:18.116266 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:18.116385 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:18.117544 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:18.117549 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:18.117553 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:46:18.117568 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:18.117591 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:46:18.117602 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:18.117692 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:18.117884 (+ 192us) server_negotiation.cc:299] Negotiation successful
0926 16:46:18.117954 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 16:46:18.119179 9222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:18.110386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34849 (local address 127.21.12.66:60661)
0926 16:46:18.111017 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:18.111061 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:18.112562 (+ 1501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:18.113150 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:18.113157 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:18.113246 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:46:18.113808 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:18.113838 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:18.114678 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:18.114683 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:18.115783 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:18.115794 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:18.116435 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:18.116439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:18.116614 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:18.117279 (+ 665us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:46:18.117281 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:46:18.117320 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:46:18.117381 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:46:18.117689 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:46:18.117695 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:46:18.117699 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 16:46:18.118222 (+ 523us) client_negotiation.cc:240] Negotiation successful
0926 16:46:18.118473 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":105,"threads_started":1}
I20260926 16:46:18.120301 9220 heartbeater.cc:347] Connected to a master server at 127.21.12.126:34849
I20260926 16:46:18.120517 9220 heartbeater.cc:464] Registering TS with master...
I20260926 16:46:18.121008 9220 heartbeater.cc:511] Master 127.21.12.126:34849 requested a full tablet report, sending...
I20260926 16:46:18.122134 8901 ts_manager.cc:194] Registered new tserver with Master: 29af901fc52f4087a777887abb50ee0a (127.21.12.66:38453)
I20260926 16:46:18.122692 8901 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.12.66:60661
W20260926 16:46:18.229235 9223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:18.229483 9223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:18.229550 9223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:18.233527 9223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:18.233577 9223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:18.233676 9223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.67
I20260926 16:46:18.238453 9223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:33761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.67
--webserver_port=0
--tserver_master_addrs=127.21.12.126:34849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.67
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:46:18.239454 9223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:18.240772 9223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:18.249863 9229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:18.249924 9228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:18.250001 9223 server_base.cc:1034] running on GCE node
W20260926 16:46:18.249963 9231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:18.250521 9223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:18.251165 9223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:18.252316 9223 hybrid_clock.cc:648] HybridClock initialized: now 1790441178252268 us; error 64 us; skew 500 ppm
I20260926 16:46:18.257644 9223 webserver.cc:460] Webserver started at http://127.21.12.67:34625/ using document root <none> and password file <none>
I20260926 16:46:18.258278 9223 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:18.258345 9223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:18.258576 9223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:46:18.260591 9223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-2/data/instance:
uuid: "eae445cc2fec4904b8fbb16c2bfbdf7c"
format_stamp: "Formatted at 2026-09-26 16:46:18 on dist-test-slave-n1hw"
server_key: "f5add8da0664427faa0ee8c5bad0efdc"
server_key_iv: "ce5dac89137605457549580ddaedebaf"
server_key_version: "encryptionkey@0"
I20260926 16:46:18.261150 9223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance:
uuid: "eae445cc2fec4904b8fbb16c2bfbdf7c"
format_stamp: "Formatted at 2026-09-26 16:46:18 on dist-test-slave-n1hw"
server_key: "f5add8da0664427faa0ee8c5bad0efdc"
server_key_iv: "ce5dac89137605457549580ddaedebaf"
server_key_version: "encryptionkey@0"
I20260926 16:46:18.265043 9223 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 16:46:18.267462 9237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:18.268666 9223 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260926 16:46:18.268777 9223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-2/wal
uuid: "eae445cc2fec4904b8fbb16c2bfbdf7c"
format_stamp: "Formatted at 2026-09-26 16:46:18 on dist-test-slave-n1hw"
server_key: "f5add8da0664427faa0ee8c5bad0efdc"
server_key_iv: "ce5dac89137605457549580ddaedebaf"
server_key_version: "encryptionkey@0"
I20260926 16:46:18.268874 9223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:18.316757 9223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:18.317492 9223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:18.317695 9223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:18.318351 9223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:46:18.319288 9223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:46:18.319332 9223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:18.319396 9223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:46:18.319428 9223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:18.327904 9223 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.67:33291
I20260926 16:46:18.327937 9351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.67:33291 every 8 connection(s)
I20260926 16:46:18.328774 9223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-2/data/info.pb
I20260926 16:46:18.334553 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 9223
I20260926 16:46:18.334649 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-2/wal/instance
I20260926 16:46:18.334945 21553 external_mini_cluster.cc:1428] Setting key df87f2f02c4e68558024c2ef90fac5f6
I20260926 16:46:18.338250 8940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:18.330556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.67:56207 (local address 127.21.12.126:34849)
0926 16:46:18.330693 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:18.330697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:18.331737 (+ 1040us) server_negotiation.cc:407] Connection header received
0926 16:46:18.333117 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:18.333121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:18.333179 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:18.333273 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:18.334199 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:18.334810 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:18.336398 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:18.336666 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:18.336773 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:18.337856 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:18.337859 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:18.337861 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:46:18.337872 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:18.337890 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:46:18.337898 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:18.337957 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:18.338087 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 16:46:18.338145 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 16:46:18.339058 9354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:18.330975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:34849 (local address 127.21.12.67:56207)
0926 16:46:18.331598 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:18.331637 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:18.332903 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:18.333435 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:18.333442 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:18.333532 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:46:18.334064 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:18.334077 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:18.334927 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:18.334935 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:18.336229 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:18.336238 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:18.336822 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:18.336828 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:18.337022 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:18.337587 (+ 565us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:46:18.337588 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:46:18.337627 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:46:18.337689 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:46:18.337967 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:46:18.337974 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:46:18.337978 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 16:46:18.338075 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 16:46:18.338288 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":116,"threads_started":1}
I20260926 16:46:18.340091 9352 heartbeater.cc:347] Connected to a master server at 127.21.12.126:34849
I20260926 16:46:18.340305 9352 heartbeater.cc:464] Registering TS with master...
I20260926 16:46:18.340809 9352 heartbeater.cc:511] Master 127.21.12.126:34849 requested a full tablet report, sending...
I20260926 16:46:18.341741 8901 ts_manager.cc:194] Registered new tserver with Master: eae445cc2fec4904b8fbb16c2bfbdf7c (127.21.12.67:33291)
I20260926 16:46:18.342319 8901 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.12.67:56207
I20260926 16:46:18.348560 21553 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:46:18.354022 8940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:18.350519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46152 (local address 127.21.12.126:34849)
0926 16:46:18.350631 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:18.350635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:18.350818 (+ 183us) server_negotiation.cc:407] Connection header received
0926 16:46:18.350916 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:18.350918 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:18.350955 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:18.351019 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:18.351284 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:18.351763 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:18.352606 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:18.352799 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:18.352870 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:18.353647 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:18.353650 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:18.353652 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:46:18.353663 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:18.353679 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:46:18.353686 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:18.353741 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:18.353877 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 16:46:18.353922 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 16:46:18.356870 8901 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46152:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:46:18.359938 8901 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:46:18.374274 9363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:18.368088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:38453 (local address 127.0.0.1:60118)
0926 16:46:18.368786 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:18.368798 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:18.368867 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:18.370140 (+ 1273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:18.370143 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:18.370160 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:46:18.370240 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:18.370246 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:18.371560 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:18.371565 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:18.372690 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:18.372700 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:18.373720 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:18.373723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:18.373862 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:18.373883 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 16:46:18.373950 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 16:46:18.374114 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"mutex_wait_us":51,"thread_start_us":94,"threads_started":1}
I20260926 16:46:18.374274 9366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:18.368365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.67:33291 (local address 127.0.0.1:57836)
0926 16:46:18.369102 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:18.369117 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:18.369205 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:18.369707 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:18.369714 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:18.369737 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:46:18.369969 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:18.369978 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:18.371101 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:18.371106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:18.372646 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:18.372659 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:18.373715 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:18.373719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:18.374003 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:18.374026 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 16:46:18.374110 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 16:46:18.374165 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"mutex_wait_us":218,"thread_start_us":92,"threads_started":1}
I20260926 16:46:18.374549 9367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:18.368790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60118 (local address 127.21.12.66:38453)
0926 16:46:18.369681 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:18.369692 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:18.369715 (+ 23us) server_negotiation.cc:407] Connection header received
0926 16:46:18.369808 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:18.369812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:18.369983 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:18.370166 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:46:18.370355 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:18.371461 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:18.372901 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:18.373607 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:18.374067 (+ 460us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:18.374240 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:18.374284 (+ 44us) server_negotiation.cc:299] Negotiation successful
0926 16:46:18.374368 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":791,"thread_start_us":110,"threads_started":1}
I20260926 16:46:18.374780 9364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:18.368002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:36101 (local address 127.0.0.1:59614)
0926 16:46:18.368605 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:18.368653 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:18.368778 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:18.369775 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:18.369779 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:18.369810 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:46:18.369938 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:18.369945 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:18.371417 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:18.371420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:18.372665 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:18.372670 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:18.373660 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:18.373664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:18.373798 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:18.373839 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 16:46:18.373962 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 16:46:18.374655 (+ 693us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":68,"spinlock_wait_cycles":2944,"thread_start_us":83,"threads_started":1}
I20260926 16:46:18.374922 9365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:18.368133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59614 (local address 127.21.12.65:36101)
0926 16:46:18.369090 (+ 957us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:18.369104 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:18.369135 (+ 31us) server_negotiation.cc:407] Connection header received
0926 16:46:18.369215 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:18.369221 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:18.369433 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:18.369594 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:46:18.370090 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:18.371303 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:18.372808 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:18.373540 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:18.373662 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:18.373892 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:18.374648 (+ 756us) server_negotiation.cc:299] Negotiation successful
0926 16:46:18.374751 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":842,"thread_start_us":102,"threads_started":1}
I20260926 16:46:18.375121 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:18.368789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57836 (local address 127.21.12.67:33291)
0926 16:46:18.369225 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:18.369236 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:18.369274 (+ 38us) server_negotiation.cc:407] Connection header received
0926 16:46:18.369349 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:18.369353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:18.369535 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:18.369944 (+ 409us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:46:18.370082 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:18.370988 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:18.372776 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:18.373564 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:18.373846 (+ 282us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:18.374047 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:18.374138 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 16:46:18.374234 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":149,"threads_started":1}
I20260926 16:46:18.377281 9154 tablet_service.cc:1467] Processing CreateTablet for tablet f6a7869565d14d0d82b1ca6c48993932 (DEFAULT_TABLE table=test-table [id=0f37898c7e9346ffbf0643c7824878fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:46:18.377552 9023 tablet_service.cc:1467] Processing CreateTablet for tablet f6a7869565d14d0d82b1ca6c48993932 (DEFAULT_TABLE table=test-table [id=0f37898c7e9346ffbf0643c7824878fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:46:18.377552 9285 tablet_service.cc:1467] Processing CreateTablet for tablet f6a7869565d14d0d82b1ca6c48993932 (DEFAULT_TABLE table=test-table [id=0f37898c7e9346ffbf0643c7824878fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:46:18.378340 9154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6a7869565d14d0d82b1ca6c48993932. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:18.378568 9285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6a7869565d14d0d82b1ca6c48993932. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:18.378568 9023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6a7869565d14d0d82b1ca6c48993932. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:18.384544 9369 tablet_bootstrap.cc:492] T f6a7869565d14d0d82b1ca6c48993932 P 29af901fc52f4087a777887abb50ee0a: Bootstrap starting.
I20260926 16:46:18.386144 9370 tablet_bootstrap.cc:492] T f6a7869565d14d0d82b1ca6c48993932 P 917f917868ee42dd9ea6977ef896a7c9: Bootstrap starting.
I20260926 16:46:18.386744 9369 tablet_bootstrap.cc:654] T f6a7869565d14d0d82b1ca6c48993932 P 29af901fc52f4087a777887abb50ee0a: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:18.387564 9369 log.cc:824] T f6a7869565d14d0d82b1ca6c48993932 P 29af901fc52f4087a777887abb50ee0a: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:18.388005 9370 tablet_bootstrap.cc:654] T f6a7869565d14d0d82b1ca6c48993932 P 917f917868ee42dd9ea6977ef896a7c9: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:18.388230 9371 tablet_bootstrap.cc:492] T f6a7869565d14d0d82b1ca6c48993932 P eae445cc2fec4904b8fbb16c2bfbdf7c: Bootstrap starting.
I20260926 16:46:18.388741 9370 log.cc:824] T f6a7869565d14d0d82b1ca6c48993932 P 917f917868ee42dd9ea6977ef896a7c9: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:18.389653 9369 tablet_bootstrap.cc:492] T f6a7869565d14d0d82b1ca6c48993932 P 29af901fc52f4087a777887abb50ee0a: No bootstrap required, opened a new log
I20260926 16:46:18.389871 9369 ts_tablet_manager.cc:1397] T f6a7869565d14d0d82b1ca6c48993932 P 29af901fc52f4087a777887abb50ee0a: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260926 16:46:18.390554 9370 tablet_bootstrap.cc:492] T f6a7869565d14d0d82b1ca6c48993932 P 917f917868ee42dd9ea6977ef896a7c9: No bootstrap required, opened a new log
I20260926 16:46:18.390745 9370 ts_tablet_manager.cc:1397] T f6a7869565d14d0d82b1ca6c48993932 P 917f917868ee42dd9ea6977ef896a7c9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 16:46:18.390754 9371 tablet_bootstrap.cc:654] T f6a7869565d14d0d82b1ca6c48993932 P eae445cc2fec4904b8fbb16c2bfbdf7c: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:18.391546 9371 log.cc:824] T f6a7869565d14d0d82b1ca6c48993932 P eae445cc2fec4904b8fbb16c2bfbdf7c: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:18.393479 9371 tablet_bootstrap.cc:492] T f6a7869565d14d0d82b1ca6c48993932 P eae445cc2fec4904b8fbb16c2bfbdf7c: No bootstrap required, opened a new log
I20260926 16:46:18.393654 9371 ts_tablet_manager.cc:1397] T f6a7869565d14d0d82b1ca6c48993932 P eae445cc2fec4904b8fbb16c2bfbdf7c: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260926 16:46:18.394091 9369 raft_consensus.cc:348] T f6a7869565d14d0d82b1ca6c48993932 P 29af901fc52f4087a777887abb50ee0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "917f917868ee42dd9ea6977ef896a7c9" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 36101 } } peers { permanent_uuid: "29af901fc52f4087a777887abb50ee0a" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 38453 } } peers { permanent_uuid: "eae445cc2fec4904b8fbb16c2bfbdf7c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33291 } }
I20260926 16:46:18.394368 9369 raft_consensus.cc:374] T f6a7869565d14d0d82b1ca6c48993932 P 29af901fc52f4087a777887abb50ee0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:46:18.394215 9370 raft_consensus.cc:348] T f6a7869565d14d0d82b1ca6c48993932 P 917f917868ee42dd9ea6977ef896a7c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "917f917868ee42dd9ea6977ef896a7c9" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 36101 } } peers { permanent_uuid: "29af901fc52f4087a777887abb50ee0a" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 38453 } } peers { permanent_uuid: "eae445cc2fec4904b8fbb16c2bfbdf7c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33291 } }
I20260926 16:46:18.394428 9369 raft_consensus.cc:729] T f6a7869565d14d0d82b1ca6c48993932 P 29af901fc52f4087a777887abb50ee0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29af901fc52f4087a777887abb50ee0a, State: Initialized, Role: FOLLOWER
I20260926 16:46:18.394439 9370 raft_consensus.cc:374] T f6a7869565d14d0d82b1ca6c48993932 P 917f917868ee42dd9ea6977ef896a7c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:46:18.394492 9370 raft_consensus.cc:729] T f6a7869565d14d0d82b1ca6c48993932 P 917f917868ee42dd9ea6977ef896a7c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 917f917868ee42dd9ea6977ef896a7c9, State: Initialized, Role: FOLLOWER
I20260926 16:46:18.394860 9369 consensus_queue.cc:260] T f6a7869565d14d0d82b1ca6c48993932 P 29af901fc52f4087a777887abb50ee0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "917f917868ee42dd9ea6977ef896a7c9" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 36101 } } peers { permanent_uuid: "29af901fc52f4087a777887abb50ee0a" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 38453 } } peers { permanent_uuid: "eae445cc2fec4904b8fbb16c2bfbdf7c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33291 } }
I20260926 16:46:18.394917 9370 consensus_queue.cc:260] T f6a7869565d14d0d82b1ca6c48993932 P 917f917868ee42dd9ea6977ef896a7c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "917f917868ee42dd9ea6977ef896a7c9" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 36101 } } peers { permanent_uuid: "29af901fc52f4087a777887abb50ee0a" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 38453 } } peers { permanent_uuid: "eae445cc2fec4904b8fbb16c2bfbdf7c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33291 } }
I20260926 16:46:18.395759 9089 heartbeater.cc:503] Master 127.21.12.126:34849 was elected leader, sending a full tablet report...
I20260926 16:46:18.395769 9220 heartbeater.cc:503] Master 127.21.12.126:34849 was elected leader, sending a full tablet report...
I20260926 16:46:18.395887 9370 ts_tablet_manager.cc:1428] T f6a7869565d14d0d82b1ca6c48993932 P 917f917868ee42dd9ea6977ef896a7c9: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 16:46:18.396889 9369 ts_tablet_manager.cc:1428] T f6a7869565d14d0d82b1ca6c48993932 P 29af901fc52f4087a777887abb50ee0a: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260926 16:46:18.398195 9371 raft_consensus.cc:348] T f6a7869565d14d0d82b1ca6c48993932 P eae445cc2fec4904b8fbb16c2bfbdf7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "917f917868ee42dd9ea6977ef896a7c9" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 36101 } } peers { permanent_uuid: "29af901fc52f4087a777887abb50ee0a" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 38453 } } peers { permanent_uuid: "eae445cc2fec4904b8fbb16c2bfbdf7c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33291 } }
I20260926 16:46:18.398411 9371 raft_consensus.cc:374] T f6a7869565d14d0d82b1ca6c48993932 P eae445cc2fec4904b8fbb16c2bfbdf7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:46:18.398523 9371 raft_consensus.cc:729] T f6a7869565d14d0d82b1ca6c48993932 P eae445cc2fec4904b8fbb16c2bfbdf7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eae445cc2fec4904b8fbb16c2bfbdf7c, State: Initialized, Role: FOLLOWER
I20260926 16:46:18.399139 9371 consensus_queue.cc:260] T f6a7869565d14d0d82b1ca6c48993932 P eae445cc2fec4904b8fbb16c2bfbdf7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "917f917868ee42dd9ea6977ef896a7c9" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 36101 } } peers { permanent_uuid: "29af901fc52f4087a777887abb50ee0a" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 38453 } } peers { permanent_uuid: "eae445cc2fec4904b8fbb16c2bfbdf7c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33291 } }
I20260926 16:46:18.399796 9352 heartbeater.cc:503] Master 127.21.12.126:34849 was elected leader, sending a full tablet report...
I20260926 16:46:18.400097 9371 ts_tablet_manager.cc:1428] T f6a7869565d14d0d82b1ca6c48993932 P eae445cc2fec4904b8fbb16c2bfbdf7c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260926 16:46:18.414326 9090 tablet.cc:2369] T f6a7869565d14d0d82b1ca6c48993932 P 917f917868ee42dd9ea6977ef896a7c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:46:18.476135 9376 raft_consensus.cc:482] T f6a7869565d14d0d82b1ca6c48993932 P 29af901fc52f4087a777887abb50ee0a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:46:18.476362 9376 raft_consensus.cc:504] T f6a7869565d14d0d82b1ca6c48993932 P 29af901fc52f4087a777887abb50ee0a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "917f917868ee42dd9ea6977ef896a7c9" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 36101 } } peers { permanent_uuid: "29af901fc52f4087a777887abb50ee0a" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 38453 } } peers { permanent_uuid: "eae445cc2fec4904b8fbb16c2bfbdf7c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33291 } }
I20260926 16:46:18.477623 9376 leader_election.cc:288] T f6a7869565d14d0d82b1ca6c48993932 P 29af901fc52f4087a777887abb50ee0a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 917f917868ee42dd9ea6977ef896a7c9 (127.21.12.65:36101), eae445cc2fec4904b8fbb16c2bfbdf7c (127.21.12.67:33291)
I20260926 16:46:18.481590 9365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:18.477969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:46295 (local address 127.21.12.65:36101)
0926 16:46:18.478109 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:18.478116 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:18.478133 (+ 17us) server_negotiation.cc:407] Connection header received
0926 16:46:18.478192 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:18.478195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:18.478244 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:18.478323 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:46:18.478649 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:18.479290 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:18.480507 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:18.481003 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:18.481095 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:18.481134 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:18.481365 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 16:46:18.481437 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 16:46:18.481635 9222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:18.477822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:36101 (local address 127.21.12.66:46295)
0926 16:46:18.477958 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:18.477974 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:18.478068 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:18.478353 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:18.478355 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:18.478376 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:46:18.478492 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:18.478499 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:18.479394 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:18.479397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:18.480373 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:18.480383 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:18.481116 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:18.481121 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:18.481223 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:18.481264 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 16:46:18.481348 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 16:46:18.481477 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 16:46:18.481823 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:18.478144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.66:33057 (local address 127.21.12.67:33291)
0926 16:46:18.478266 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:18.478272 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:18.478558 (+ 286us) server_negotiation.cc:407] Connection header received
0926 16:46:18.478643 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:18.478646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:18.478701 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:18.478766 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:46:18.479039 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:18.479622 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:18.480708 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:18.481208 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:18.481350 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:18.481402 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:18.481587 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 16:46:18.481670 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 16:46:18.481927 9378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:18.477988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.67:33291 (local address 127.21.12.66:33057)
0926 16:46:18.478463 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:18.478474 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:18.478551 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:18.478796 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:18.478799 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:18.478815 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:46:18.478926 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:18.478932 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:18.479741 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:18.479743 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:18.480581 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:18.480590 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:18.481350 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:18.481353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:18.481454 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:18.481475 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 16:46:18.481551 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 16:46:18.481609 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"mutex_wait_us":1,"thread_start_us":136,"threads_started":1}
I20260926 16:46:18.482420 9043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6a7869565d14d0d82b1ca6c48993932" candidate_uuid: "29af901fc52f4087a777887abb50ee0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "917f917868ee42dd9ea6977ef896a7c9" is_pre_election: true
I20260926 16:46:18.482547 9305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6a7869565d14d0d82b1ca6c48993932" candidate_uuid: "29af901fc52f4087a777887abb50ee0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eae445cc2fec4904b8fbb16c2bfbdf7c" is_pre_election: true
I20260926 16:46:18.482760 9043 raft_consensus.cc:2454] T f6a7869565d14d0d82b1ca6c48993932 P 917f917868ee42dd9ea6977ef896a7c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29af901fc52f4087a777887abb50ee0a in term 0.
I20260926 16:46:18.482779 9305 raft_consensus.cc:2454] T f6a7869565d14d0d82b1ca6c48993932 P eae445cc2fec4904b8fbb16c2bfbdf7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29af901fc52f4087a777887abb50ee0a in term 0.
I20260926 16:46:18.483345 9108 leader_election.cc:302] T f6a7869565d14d0d82b1ca6c48993932 P 29af901fc52f4087a777887abb50ee0a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 29af901fc52f4087a777887abb50ee0a, 917f917868ee42dd9ea6977ef896a7c9, eae445cc2fec4904b8fbb16c2bfbdf7c; no voters:
I20260926 16:46:18.483642 9376 raft_consensus.cc:2781] T f6a7869565d14d0d82b1ca6c48993932 P 29af901fc52f4087a777887abb50ee0a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:46:18.483742 9376 raft_consensus.cc:482] T f6a7869565d14d0d82b1ca6c48993932 P 29af901fc52f4087a777887abb50ee0a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:46:18.483803 9376 raft_consensus.cc:3037] T f6a7869565d14d0d82b1ca6c48993932 P 29af901fc52f4087a777887abb50ee0a [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:18.484831 9376 raft_consensus.cc:504] T f6a7869565d14d0d82b1ca6c48993932 P 29af901fc52f4087a777887abb50ee0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "917f917868ee42dd9ea6977ef896a7c9" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 36101 } } peers { permanent_uuid: "29af901fc52f4087a777887abb50ee0a" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 38453 } } peers { permanent_uuid: "eae445cc2fec4904b8fbb16c2bfbdf7c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33291 } }
I20260926 16:46:18.485186 9376 leader_election.cc:288] T f6a7869565d14d0d82b1ca6c48993932 P 29af901fc52f4087a777887abb50ee0a [CANDIDATE]: Term 1 election: Requested vote from peers 917f917868ee42dd9ea6977ef896a7c9 (127.21.12.65:36101), eae445cc2fec4904b8fbb16c2bfbdf7c (127.21.12.67:33291)
I20260926 16:46:18.485668 9043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6a7869565d14d0d82b1ca6c48993932" candidate_uuid: "29af901fc52f4087a777887abb50ee0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "917f917868ee42dd9ea6977ef896a7c9"
I20260926 16:46:18.485700 9305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6a7869565d14d0d82b1ca6c48993932" candidate_uuid: "29af901fc52f4087a777887abb50ee0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eae445cc2fec4904b8fbb16c2bfbdf7c"
I20260926 16:46:18.485848 9043 raft_consensus.cc:3037] T f6a7869565d14d0d82b1ca6c48993932 P 917f917868ee42dd9ea6977ef896a7c9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:18.485850 9305 raft_consensus.cc:3037] T f6a7869565d14d0d82b1ca6c48993932 P eae445cc2fec4904b8fbb16c2bfbdf7c [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:18.487164 9305 raft_consensus.cc:2454] T f6a7869565d14d0d82b1ca6c48993932 P eae445cc2fec4904b8fbb16c2bfbdf7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29af901fc52f4087a777887abb50ee0a in term 1.
I20260926 16:46:18.487164 9043 raft_consensus.cc:2454] T f6a7869565d14d0d82b1ca6c48993932 P 917f917868ee42dd9ea6977ef896a7c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29af901fc52f4087a777887abb50ee0a in term 1.
I20260926 16:46:18.487460 9110 leader_election.cc:302] T f6a7869565d14d0d82b1ca6c48993932 P 29af901fc52f4087a777887abb50ee0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29af901fc52f4087a777887abb50ee0a, 917f917868ee42dd9ea6977ef896a7c9; no voters:
I20260926 16:46:18.487653 9376 raft_consensus.cc:2781] T f6a7869565d14d0d82b1ca6c48993932 P 29af901fc52f4087a777887abb50ee0a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:46:18.487912 9376 raft_consensus.cc:686] T f6a7869565d14d0d82b1ca6c48993932 P 29af901fc52f4087a777887abb50ee0a [term 1 LEADER]: Becoming Leader. State: Replica: 29af901fc52f4087a777887abb50ee0a, State: Running, Role: LEADER
I20260926 16:46:18.488235 9376 consensus_queue.cc:237] T f6a7869565d14d0d82b1ca6c48993932 P 29af901fc52f4087a777887abb50ee0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "917f917868ee42dd9ea6977ef896a7c9" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 36101 } } peers { permanent_uuid: "29af901fc52f4087a777887abb50ee0a" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 38453 } } peers { permanent_uuid: "eae445cc2fec4904b8fbb16c2bfbdf7c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33291 } }
I20260926 16:46:18.490939 8896 catalog_manager.cc:5495] T f6a7869565d14d0d82b1ca6c48993932 P 29af901fc52f4087a777887abb50ee0a reported cstate change: term changed from 0 to 1, leader changed from <none> to 29af901fc52f4087a777887abb50ee0a (127.21.12.66). New cstate: current_term: 1 leader_uuid: "29af901fc52f4087a777887abb50ee0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "917f917868ee42dd9ea6977ef896a7c9" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 36101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29af901fc52f4087a777887abb50ee0a" member_type: VOTER last_known_addr { host: "127.21.12.66" port: 38453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eae445cc2fec4904b8fbb16c2bfbdf7c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33291 } health_report { overall_health: UNKNOWN } } }
I20260926 16:46:18.525571 9367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:18.522113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60120 (local address 127.21.12.66:38453)
0926 16:46:18.522256 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:18.522262 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:18.522279 (+ 17us) server_negotiation.cc:407] Connection header received
0926 16:46:18.522360 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:18.522362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:18.522415 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:18.522523 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:46:18.522799 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:18.523297 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:18.524156 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:18.524368 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:18.524474 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:18.524771 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:46:18.525187 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:46:18.525319 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:18.525360 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 16:46:18.525409 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 16:46:18.531629 9305 raft_consensus.cc:1261] T f6a7869565d14d0d82b1ca6c48993932 P eae445cc2fec4904b8fbb16c2bfbdf7c [term 1 FOLLOWER]: Refusing update from remote peer 29af901fc52f4087a777887abb50ee0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:46:18.531633 9043 raft_consensus.cc:1261] T f6a7869565d14d0d82b1ca6c48993932 P 917f917868ee42dd9ea6977ef896a7c9 [term 1 FOLLOWER]: Refusing update from remote peer 29af901fc52f4087a777887abb50ee0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:46:18.532316 9376 consensus_queue.cc:1035] T f6a7869565d14d0d82b1ca6c48993932 P 29af901fc52f4087a777887abb50ee0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae445cc2fec4904b8fbb16c2bfbdf7c" member_type: VOTER last_known_addr { host: "127.21.12.67" port: 33291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:46:18.532491 9379 consensus_queue.cc:1035] T f6a7869565d14d0d82b1ca6c48993932 P 29af901fc52f4087a777887abb50ee0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "917f917868ee42dd9ea6977ef896a7c9" member_type: VOTER last_known_addr { host: "127.21.12.65" port: 36101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:46:18.540652 9384 mvcc.cc:205] Tried to move back new op lower bound from 7333647067257868288 to 7333647067090186240. Current Snapshot: MvccSnapshot[applied={T|T < 7333647067257868288}]
I20260926 16:46:18.541141 9382 mvcc.cc:205] Tried to move back new op lower bound from 7333647067257868288 to 7333647067090186240. Current Snapshot: MvccSnapshot[applied={T|T < 7333647067257868288}]
I20260926 16:46:18.550117 8896 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:46:18.550318 8896 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:46:18.552858 8896 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f9e6b25430044db382aae127ad7a72a9: Sending DeleteTablet for 3 replicas of tablet f6a7869565d14d0d82b1ca6c48993932
I20260926 16:46:18.553594 9154 tablet_service.cc:1514] Processing DeleteTablet for tablet f6a7869565d14d0d82b1ca6c48993932 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:46:18 UTC) from {username='slave'} at 127.0.0.1:60118
I20260926 16:46:18.553606 9023 tablet_service.cc:1514] Processing DeleteTablet for tablet f6a7869565d14d0d82b1ca6c48993932 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:46:18 UTC) from {username='slave'} at 127.0.0.1:59614
I20260926 16:46:18.553848 9285 tablet_service.cc:1514] Processing DeleteTablet for tablet f6a7869565d14d0d82b1ca6c48993932 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:46:18 UTC) from {username='slave'} at 127.0.0.1:57836
I20260926 16:46:18.554103 9392 tablet_replica.cc:330] T f6a7869565d14d0d82b1ca6c48993932 P 29af901fc52f4087a777887abb50ee0a: stopping tablet replica
I20260926 16:46:18.554447 9393 tablet_replica.cc:330] T f6a7869565d14d0d82b1ca6c48993932 P 917f917868ee42dd9ea6977ef896a7c9: stopping tablet replica
I20260926 16:46:18.554519 9394 tablet_replica.cc:330] T f6a7869565d14d0d82b1ca6c48993932 P eae445cc2fec4904b8fbb16c2bfbdf7c: stopping tablet replica
I20260926 16:46:18.554769 9393 raft_consensus.cc:2229] T f6a7869565d14d0d82b1ca6c48993932 P 917f917868ee42dd9ea6977ef896a7c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:46:18.554829 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 8961
I20260926 16:46:18.554821 9394 raft_consensus.cc:2229] T f6a7869565d14d0d82b1ca6c48993932 P eae445cc2fec4904b8fbb16c2bfbdf7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:46:18.557394 9394 raft_consensus.cc:2258] T f6a7869565d14d0d82b1ca6c48993932 P eae445cc2fec4904b8fbb16c2bfbdf7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:46:18.558682 9392 raft_consensus.cc:2229] T f6a7869565d14d0d82b1ca6c48993932 P 29af901fc52f4087a777887abb50ee0a [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:46:18.559109 9392 raft_consensus.cc:2258] T f6a7869565d14d0d82b1ca6c48993932 P 29af901fc52f4087a777887abb50ee0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:46:18.559677 9394 ts_tablet_manager.cc:1905] T f6a7869565d14d0d82b1ca6c48993932 P eae445cc2fec4904b8fbb16c2bfbdf7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:46:18.560366 9392 ts_tablet_manager.cc:1905] T f6a7869565d14d0d82b1ca6c48993932 P 29af901fc52f4087a777887abb50ee0a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 16:46:18.562294 8888 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.12.65:36101 (error 108)
I20260926 16:46:18.562728 9394 ts_tablet_manager.cc:1918] T f6a7869565d14d0d82b1ca6c48993932 P eae445cc2fec4904b8fbb16c2bfbdf7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:46:18.562899 9394 log.cc:1196] T f6a7869565d14d0d82b1ca6c48993932 P eae445cc2fec4904b8fbb16c2bfbdf7c: Deleting WAL directory at /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440924937351-21553-0/minicluster-data/ts-2/wal/wals/f6a7869565d14d0d82b1ca6c48993932
I20260926 16:46:18.563129 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 9092
I20260926 16:46:18.563349 9394 ts_tablet_manager.cc:1939] T f6a7869565d14d0d82b1ca6c48993932 P eae445cc2fec4904b8fbb16c2bfbdf7c: Deleting consensus metadata
I20260926 16:46:18.564774 8886 catalog_manager.cc:4841] TS eae445cc2fec4904b8fbb16c2bfbdf7c (127.21.12.67:33291): tablet f6a7869565d14d0d82b1ca6c48993932 (table test-table [id=0f37898c7e9346ffbf0643c7824878fe]) successfully deleted
I20260926 16:46:18.565294 9364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:18.564901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.65:36101 (local address 127.0.0.1:59628)
0926 16:46:18.565162 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:18.565223 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.65:36101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
W20260926 16:46:18.565460 8888 catalog_manager.cc:4584] TS 917f917868ee42dd9ea6977ef896a7c9 (127.21.12.65:36101): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f6a7869565d14d0d82b1ca6c48993932: Network error: Client connection negotiation failed: client connection to 127.21.12.65:36101: connect: Connection refused (error 111)
I20260926 16:46:18.569373 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 9223
I20260926 16:46:18.569965 9364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:18.569617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.66:38453 (local address 127.0.0.1:60126)
0926 16:46:18.569816 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:18.569870 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.66:38453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260926 16:46:18.575500 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 8870
2026-09-26T16:46:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1113 ms)
[----------] 6 tests from EncryptionPolicyTest (9876 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440924937351-21553-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:46:18 dist-test-slave-n1hw krb5kdc[9398](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:46:18 dist-test-slave-n1hw krb5kdc[9398](info): set up 2 sockets
Sep 26 16:46:18 dist-test-slave-n1hw krb5kdc[9398](info): commencing operation
krb5kdc: starting...
W20260926 16:46:19.623946 21553 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Sep 26 16:46:19 dist-test-slave-n1hw krb5kdc[9398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:46:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:46:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440924937351-21553-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440924937351-21553-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 16:46:19.744742 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39705
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440924937351-21553-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:46:19.852917 9411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:19.853221 9411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:19.853295 9411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:19.857290 9411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:46:19.857384 9411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:19.857424 9411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:19.857477 9411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:46:19.857513 9411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:46:19.862773 9411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39705
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440924937351-21553-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:46:19.863842 9411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:19.864864 9411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:19.873239 9419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:19.873430 9411 server_base.cc:1034] running on GCE node
W20260926 16:46:19.873239 9417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:19.873270 9416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:19.874125 9411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:19.875150 9411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:19.876307 9411 hybrid_clock.cc:648] HybridClock initialized: now 1790441179876294 us; error 24 us; skew 500 ppm
Sep 26 16:46:19 dist-test-slave-n1hw krb5kdc[9398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:46:19.878985 9411 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 16:46:19.879854 9411 webserver.cc:460] Webserver started at http://127.0.0.1:33945/ using document root <none> and password file <none>
I20260926 16:46:19.880391 9411 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:19.880475 9411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:19.880723 9411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:46:19.882771 9411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "a4fe79555f304c99a98b8e1a7bf1d12e"
format_stamp: "Formatted at 2026-09-26 16:46:19 on dist-test-slave-n1hw"
server_key: "65f224b221e76dee2e804f76a74222c3"
server_key_iv: "08b9b2b5777ce78c6ba8e6431c4a5211"
server_key_version: "encryptionkey@0"
I20260926 16:46:19.883323 9411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "a4fe79555f304c99a98b8e1a7bf1d12e"
format_stamp: "Formatted at 2026-09-26 16:46:19 on dist-test-slave-n1hw"
server_key: "65f224b221e76dee2e804f76a74222c3"
server_key_iv: "08b9b2b5777ce78c6ba8e6431c4a5211"
server_key_version: "encryptionkey@0"
I20260926 16:46:19.887102 9411 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 16:46:19.889539 9426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:19.890851 9411 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:46:19.890985 9411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "a4fe79555f304c99a98b8e1a7bf1d12e"
format_stamp: "Formatted at 2026-09-26 16:46:19 on dist-test-slave-n1hw"
server_key: "65f224b221e76dee2e804f76a74222c3"
server_key_iv: "08b9b2b5777ce78c6ba8e6431c4a5211"
server_key_version: "encryptionkey@0"
I20260926 16:46:19.891096 9411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:19.905341 9411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:19.908289 9411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:19.908492 9411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:19.915577 9411 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39705
I20260926 16:46:19.915644 9479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39705 every 8 connection(s)
I20260926 16:46:19.916538 9411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:46:19.919461 9480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:19.920627 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 9411
I20260926 16:46:19.920728 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:46:19.921030 21553 external_mini_cluster.cc:1428] Setting key 4fd80e980bcd47c404aa655c8d6808e9
I20260926 16:46:19.925938 9480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4fe79555f304c99a98b8e1a7bf1d12e: Bootstrap starting.
I20260926 16:46:19.928673 9480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4fe79555f304c99a98b8e1a7bf1d12e: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:19.929641 9480 log.cc:824] T 00000000000000000000000000000000 P a4fe79555f304c99a98b8e1a7bf1d12e: Log is configured to *not* fsync() on all Append() calls
Sep 26 16:46:19 dist-test-slave-n1hw krb5kdc[9398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 16:46:19.931874 9480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4fe79555f304c99a98b8e1a7bf1d12e: No bootstrap required, opened a new log
I20260926 16:46:19.935729 9483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:19.922369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38887 (local address 127.0.0.1:39705)
0926 16:46:19.922755 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:19.922776 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:19.922811 (+ 35us) server_negotiation.cc:407] Connection header received
0926 16:46:19.923424 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:19.923440 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:19.923773 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:19.924045 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:19.925057 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:19.926158 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:19.927337 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:19.927780 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:19.927891 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:19.931051 (+ 3160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:19.931057 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:19.931069 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:19.931094 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:19.932835 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:19.933376 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:19.933381 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:19.933385 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:19.933503 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:19.933846 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:19.933848 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:19.933850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:19.934225 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:19.934380 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:19.934749 (+ 369us) server_negotiation.cc:299] Negotiation successful
0926 16:46:19.934993 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":126,"threads_started":1}
I20260926 16:46:19.936224 9480 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4fe79555f304c99a98b8e1a7bf1d12e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4fe79555f304c99a98b8e1a7bf1d12e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39705 } }
I20260926 16:46:19.936496 9480 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4fe79555f304c99a98b8e1a7bf1d12e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:46:19.936542 9480 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4fe79555f304c99a98b8e1a7bf1d12e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4fe79555f304c99a98b8e1a7bf1d12e, State: Initialized, Role: FOLLOWER
I20260926 16:46:19.936996 9480 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4fe79555f304c99a98b8e1a7bf1d12e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4fe79555f304c99a98b8e1a7bf1d12e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39705 } }
I20260926 16:46:19.937129 9480 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4fe79555f304c99a98b8e1a7bf1d12e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:46:19.937202 9480 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4fe79555f304c99a98b8e1a7bf1d12e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:46:19.937314 9480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4fe79555f304c99a98b8e1a7bf1d12e [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:19.938441 9480 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4fe79555f304c99a98b8e1a7bf1d12e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4fe79555f304c99a98b8e1a7bf1d12e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39705 } }
I20260926 16:46:19.938794 9480 leader_election.cc:302] T 00000000000000000000000000000000 P a4fe79555f304c99a98b8e1a7bf1d12e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4fe79555f304c99a98b8e1a7bf1d12e; no voters:
I20260926 16:46:19.938995 21553 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:46:19.939169 9480 leader_election.cc:288] T 00000000000000000000000000000000 P a4fe79555f304c99a98b8e1a7bf1d12e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:46:19.939280 9485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4fe79555f304c99a98b8e1a7bf1d12e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:46:19.939476 9485 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4fe79555f304c99a98b8e1a7bf1d12e [term 1 LEADER]: Becoming Leader. State: Replica: a4fe79555f304c99a98b8e1a7bf1d12e, State: Running, Role: LEADER
I20260926 16:46:19.939801 9485 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4fe79555f304c99a98b8e1a7bf1d12e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4fe79555f304c99a98b8e1a7bf1d12e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39705 } }
I20260926 16:46:19.940434 9480 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4fe79555f304c99a98b8e1a7bf1d12e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:46:19.941028 9485 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4fe79555f304c99a98b8e1a7bf1d12e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4fe79555f304c99a98b8e1a7bf1d12e. Latest consensus state: current_term: 1 leader_uuid: "a4fe79555f304c99a98b8e1a7bf1d12e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4fe79555f304c99a98b8e1a7bf1d12e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39705 } } }
I20260926 16:46:19.941151 9485 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4fe79555f304c99a98b8e1a7bf1d12e [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:19.941519 9496 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:46:19.941463 9486 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4fe79555f304c99a98b8e1a7bf1d12e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4fe79555f304c99a98b8e1a7bf1d12e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4fe79555f304c99a98b8e1a7bf1d12e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39705 } } }
I20260926 16:46:19.941602 9486 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4fe79555f304c99a98b8e1a7bf1d12e [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:19.944664 9496 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:46:19.951014 9496 catalog_manager.cc:1347] Generated new cluster ID: 3b314bd0dc7e4aa89aad3ca0f2513a01
I20260926 16:46:19.951079 9496 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:46:19.951797 9483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:19.941242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57031 (local address 127.0.0.1:39705)
0926 16:46:19.941362 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:19.941368 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:19.942681 (+ 1313us) server_negotiation.cc:407] Connection header received
0926 16:46:19.942844 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:19.942849 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:19.942895 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:19.942948 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:19.943908 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:19.944628 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:19.945747 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:19.946144 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:19.946255 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:19.947757 (+ 1502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:19.947761 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:19.947764 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:19.947789 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:19.949417 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:19.949996 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:19.950001 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:19.950004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:19.950105 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:19.950470 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:19.950474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:19.950476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:19.950686 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:19.950832 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:19.951533 (+ 701us) server_negotiation.cc:299] Negotiation successful
0926 16:46:19.951633 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 16:46:19.982482 9496 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:46:19.983464 9496 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:46:19.992177 9496 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4fe79555f304c99a98b8e1a7bf1d12e: Generated new TSK 0
I20260926 16:46:19.992919 9496 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:46:20.006757 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 9411
W20260926 16:46:20.012305 9396 connection.cc:535] client connection to 127.0.0.1:39705 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39705): Connection reset by peer (error 104)
2026-09-26T16:46:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1429 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440924937351-21553-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:46:20 dist-test-slave-n1hw krb5kdc[9511](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:46:20 dist-test-slave-n1hw krb5kdc[9511](info): set up 2 sockets
Sep 26 16:46:20 dist-test-slave-n1hw krb5kdc[9511](info): commencing operation
krb5kdc: starting...
W20260926 16:46:21.056131 21553 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Sep 26 16:46:21 dist-test-slave-n1hw krb5kdc[9511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:46:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:46:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440924937351-21553-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440924937351-21553-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 16:46:21.177146 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41593
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440924937351-21553-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:46:21.286432 9524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:21.286747 9524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:21.286839 9524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:21.290956 9524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:46:21.291038 9524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:21.291079 9524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:21.291116 9524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:46:21.291149 9524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:46:21.296324 9524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41593
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440924937351-21553-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:46:21.297371 9524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:21.298413 9524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:21.306880 9532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:21.307034 9524 server_base.cc:1034] running on GCE node
W20260926 16:46:21.306879 9530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:21.306924 9529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:21.307667 9524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:21.308779 9524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:21.309967 9524 hybrid_clock.cc:648] HybridClock initialized: now 1790441181309948 us; error 31 us; skew 500 ppm
Sep 26 16:46:21 dist-test-slave-n1hw krb5kdc[9511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:46:21.312808 9524 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 16:46:21.313665 9524 webserver.cc:460] Webserver started at http://127.0.0.1:44857/ using document root <none> and password file <none>
I20260926 16:46:21.314229 9524 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:21.314291 9524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:21.314452 9524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:46:21.316398 9524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "0cb3d135b35a4301a45c22f550b26229"
format_stamp: "Formatted at 2026-09-26 16:46:21 on dist-test-slave-n1hw"
server_key: "04b52cf6c70655a19cac548989a990e8"
server_key_iv: "b49dbe45ca45cc7cfee960b8484383c5"
server_key_version: "encryptionkey@0"
I20260926 16:46:21.316911 9524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "0cb3d135b35a4301a45c22f550b26229"
format_stamp: "Formatted at 2026-09-26 16:46:21 on dist-test-slave-n1hw"
server_key: "04b52cf6c70655a19cac548989a990e8"
server_key_iv: "b49dbe45ca45cc7cfee960b8484383c5"
server_key_version: "encryptionkey@0"
I20260926 16:46:21.320601 9524 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260926 16:46:21.323028 9539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:21.324301 9524 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 16:46:21.324437 9524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "0cb3d135b35a4301a45c22f550b26229"
format_stamp: "Formatted at 2026-09-26 16:46:21 on dist-test-slave-n1hw"
server_key: "04b52cf6c70655a19cac548989a990e8"
server_key_iv: "b49dbe45ca45cc7cfee960b8484383c5"
server_key_version: "encryptionkey@0"
I20260926 16:46:21.324538 9524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:21.337908 9524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:21.341049 9524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:21.341269 9524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:21.348348 9524 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41593
I20260926 16:46:21.348372 9592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41593 every 8 connection(s)
I20260926 16:46:21.349268 9524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:46:21.351979 9593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:21.352769 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 9524
I20260926 16:46:21.352905 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:46:21.353188 21553 external_mini_cluster.cc:1428] Setting key 2e9f06dced2c7f8bb6867ea3a383bac2
I20260926 16:46:21.357939 9593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cb3d135b35a4301a45c22f550b26229: Bootstrap starting.
I20260926 16:46:21.360107 9593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0cb3d135b35a4301a45c22f550b26229: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:21.360816 9593 log.cc:824] T 00000000000000000000000000000000 P 0cb3d135b35a4301a45c22f550b26229: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:21.362088 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:21.354575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39059 (local address 127.0.0.1:41593)
0926 16:46:21.354991 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:21.355005 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:21.355039 (+ 34us) server_negotiation.cc:407] Connection header received
0926 16:46:21.355679 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:21.355696 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:21.356031 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:21.356358 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:21.356761 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:21.357693 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:21.358679 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:21.359134 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:21.359297 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:21.360198 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:21.360220 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:21.360239 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:46:21.360647 (+ 408us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:21.360700 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:46:21.360722 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:21.360860 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:21.361044 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 16:46:21.361173 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":119,"threads_started":1}
I20260926 16:46:21.363096 9593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cb3d135b35a4301a45c22f550b26229: No bootstrap required, opened a new log
I20260926 16:46:21.365113 21553 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:46:21.367321 9593 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cb3d135b35a4301a45c22f550b26229 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cb3d135b35a4301a45c22f550b26229" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41593 } }
I20260926 16:46:21.367575 9593 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0cb3d135b35a4301a45c22f550b26229 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:46:21.367660 9593 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cb3d135b35a4301a45c22f550b26229 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cb3d135b35a4301a45c22f550b26229, State: Initialized, Role: FOLLOWER
I20260926 16:46:21.368155 9593 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cb3d135b35a4301a45c22f550b26229 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cb3d135b35a4301a45c22f550b26229" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41593 } }
I20260926 16:46:21.368332 9593 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0cb3d135b35a4301a45c22f550b26229 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:46:21.368405 9593 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cb3d135b35a4301a45c22f550b26229 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:46:21.368505 9593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cb3d135b35a4301a45c22f550b26229 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:21.369577 9593 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cb3d135b35a4301a45c22f550b26229 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cb3d135b35a4301a45c22f550b26229" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41593 } }
I20260926 16:46:21.369987 9593 leader_election.cc:302] T 00000000000000000000000000000000 P 0cb3d135b35a4301a45c22f550b26229 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0cb3d135b35a4301a45c22f550b26229; no voters:
I20260926 16:46:21.370373 9593 leader_election.cc:288] T 00000000000000000000000000000000 P 0cb3d135b35a4301a45c22f550b26229 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:46:21.370868 9603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cb3d135b35a4301a45c22f550b26229 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:46:21.370846 9604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cb3d135b35a4301a45c22f550b26229 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cb3d135b35a4301a45c22f550b26229" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41593 } } }
I20260926 16:46:21.371083 9604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cb3d135b35a4301a45c22f550b26229 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:46:21.371135 9603 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0cb3d135b35a4301a45c22f550b26229 [term 1 LEADER]: Becoming Leader. State: Replica: 0cb3d135b35a4301a45c22f550b26229, State: Running, Role: LEADER
I20260926 16:46:21.371469 9603 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0cb3d135b35a4301a45c22f550b26229 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cb3d135b35a4301a45c22f550b26229" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41593 } }
I20260926 16:46:21.371745 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:21.367074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46737 (local address 127.0.0.1:41593)
0926 16:46:21.367203 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:21.367207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:21.367415 (+ 208us) server_negotiation.cc:407] Connection header received
0926 16:46:21.367532 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:21.367537 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:21.367601 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:21.367679 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:21.367979 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:21.368755 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:21.369577 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:21.369998 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:21.370134 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:21.371088 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:21.371092 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:21.371095 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:46:21.371110 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:21.371138 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:46:21.371161 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:21.371251 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:21.371570 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 16:46:21.371608 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 16:46:21.371833 9593 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cb3d135b35a4301a45c22f550b26229 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:46:21.373135 9604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cb3d135b35a4301a45c22f550b26229 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cb3d135b35a4301a45c22f550b26229. Latest consensus state: current_term: 1 leader_uuid: "0cb3d135b35a4301a45c22f550b26229" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cb3d135b35a4301a45c22f550b26229" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41593 } } }
I20260926 16:46:21.373210 9604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cb3d135b35a4301a45c22f550b26229 [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:21.373548 9612 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:46:21.376324 9612 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:46:21.381695 9612 catalog_manager.cc:1347] Generated new cluster ID: 75487546a15a437093ce95fedd3a854d
I20260926 16:46:21.381774 9612 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:46:21.404359 9612 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:46:21.405668 9612 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:46:21.412585 9612 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0cb3d135b35a4301a45c22f550b26229: Generated new TSK 0
I20260926 16:46:21.413244 9612 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:46:21.423194 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 9524
W20260926 16:46:21.428555 9509 connection.cc:535] client connection to 127.0.0.1:41593 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41593): Connection reset by peer (error 104)
2026-09-26T16:46:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1416 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440924937351-21553-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:46:21 dist-test-slave-n1hw krb5kdc[9624](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:46:21 dist-test-slave-n1hw krb5kdc[9624](info): set up 2 sockets
Sep 26 16:46:21 dist-test-slave-n1hw krb5kdc[9624](info): commencing operation
krb5kdc: starting...
W20260926 16:46:22.464035 21553 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Sep 26 16:46:22 dist-test-slave-n1hw krb5kdc[9624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:46:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:46:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440924937351-21553-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440924937351-21553-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 16:46:22.585410 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38427
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440924937351-21553-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38427
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:46:22.694729 9637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:22.694983 9637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:22.695024 9637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:22.698871 9637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:46:22.698920 9637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:22.698937 9637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:22.698953 9637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:46:22.698968 9637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:46:22.704118 9637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38427
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440924937351-21553-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:46:22.705047 9637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:22.705988 9637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:22.714284 9645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:22.714284 9642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:22.714366 9637 server_base.cc:1034] running on GCE node
W20260926 16:46:22.714311 9643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:22.715137 9637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:22.716171 9637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:22.717373 9637 hybrid_clock.cc:648] HybridClock initialized: now 1790441182717342 us; error 42 us; skew 500 ppm
Sep 26 16:46:22 dist-test-slave-n1hw krb5kdc[9624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:46:22.720316 9637 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 16:46:22.721271 9637 webserver.cc:460] Webserver started at http://127.0.0.1:36065/ using document root <none> and password file <none>
I20260926 16:46:22.721827 9637 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:22.721892 9637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:22.722065 9637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:46:22.724112 9637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "ee259c731e28486691317940ac6b211e"
format_stamp: "Formatted at 2026-09-26 16:46:22 on dist-test-slave-n1hw"
server_key: "bbb8614abf10a10f900a9b4fe24c7a6c"
server_key_iv: "a70a8efd63df514ed2b0e9e123a54081"
server_key_version: "encryptionkey@0"
I20260926 16:46:22.724615 9637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "ee259c731e28486691317940ac6b211e"
format_stamp: "Formatted at 2026-09-26 16:46:22 on dist-test-slave-n1hw"
server_key: "bbb8614abf10a10f900a9b4fe24c7a6c"
server_key_iv: "a70a8efd63df514ed2b0e9e123a54081"
server_key_version: "encryptionkey@0"
I20260926 16:46:22.728358 9637 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 16:46:22.730753 9652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:22.732071 9637 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:46:22.732214 9637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "ee259c731e28486691317940ac6b211e"
format_stamp: "Formatted at 2026-09-26 16:46:22 on dist-test-slave-n1hw"
server_key: "bbb8614abf10a10f900a9b4fe24c7a6c"
server_key_iv: "a70a8efd63df514ed2b0e9e123a54081"
server_key_version: "encryptionkey@0"
I20260926 16:46:22.732323 9637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:22.737011 9637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:22.740247 9637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:22.740438 9637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:22.747404 9637 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38427
I20260926 16:46:22.747427 9705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38427 every 8 connection(s)
I20260926 16:46:22.748296 9637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:46:22.751022 9706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:22.751169 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 9637
I20260926 16:46:22.751319 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:46:22.751667 21553 external_mini_cluster.cc:1428] Setting key 91924b60953a8b25ba20b165c8665046
I20260926 16:46:22.756831 9709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:22.752908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39783 (local address 127.0.0.1:38427)
0926 16:46:22.753301 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:22.753315 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:22.753348 (+ 33us) server_negotiation.cc:407] Connection header received
0926 16:46:22.754081 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:22.754098 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:22.754367 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:22.754636 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:22.755530 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:22.755535 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:22.755549 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:46:22.755560 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:22.755655 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:46:22.755680 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:22.755814 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:22.755912 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 16:46:22.756106 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":127,"threads_started":1}
I20260926 16:46:22.757730 9706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee259c731e28486691317940ac6b211e: Bootstrap starting.
I20260926 16:46:22.759723 21553 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:46:22.760236 9706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee259c731e28486691317940ac6b211e: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:22.760931 9706 log.cc:824] T 00000000000000000000000000000000 P ee259c731e28486691317940ac6b211e: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:22.763192 9706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee259c731e28486691317940ac6b211e: No bootstrap required, opened a new log
I20260926 16:46:22.763816 9709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:22.762018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54075 (local address 127.0.0.1:38427)
0926 16:46:22.762138 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:22.762143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:22.762337 (+ 194us) server_negotiation.cc:407] Connection header received
0926 16:46:22.762424 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:22.762427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:22.762462 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:22.762600 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:22.763441 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:22.763444 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:22.763447 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:46:22.763459 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:22.763486 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:46:22.763499 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:22.763605 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:22.763668 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 16:46:22.763705 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 16:46:22.766939 9706 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee259c731e28486691317940ac6b211e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee259c731e28486691317940ac6b211e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38427 } }
I20260926 16:46:22.767161 9706 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee259c731e28486691317940ac6b211e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:46:22.767194 9706 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee259c731e28486691317940ac6b211e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee259c731e28486691317940ac6b211e, State: Initialized, Role: FOLLOWER
I20260926 16:46:22.767549 9706 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee259c731e28486691317940ac6b211e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee259c731e28486691317940ac6b211e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38427 } }
I20260926 16:46:22.767653 9706 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee259c731e28486691317940ac6b211e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:46:22.767699 9706 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee259c731e28486691317940ac6b211e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:46:22.767769 9706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee259c731e28486691317940ac6b211e [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:22.768574 9706 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee259c731e28486691317940ac6b211e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee259c731e28486691317940ac6b211e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38427 } }
I20260926 16:46:22.768862 9706 leader_election.cc:302] T 00000000000000000000000000000000 P ee259c731e28486691317940ac6b211e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee259c731e28486691317940ac6b211e; no voters:
I20260926 16:46:22.769121 9706 leader_election.cc:288] T 00000000000000000000000000000000 P ee259c731e28486691317940ac6b211e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:46:22.769253 9716 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee259c731e28486691317940ac6b211e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:46:22.769516 9716 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee259c731e28486691317940ac6b211e [term 1 LEADER]: Becoming Leader. State: Replica: ee259c731e28486691317940ac6b211e, State: Running, Role: LEADER
I20260926 16:46:22.769936 9716 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee259c731e28486691317940ac6b211e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee259c731e28486691317940ac6b211e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38427 } }
I20260926 16:46:22.770361 9706 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee259c731e28486691317940ac6b211e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:46:22.771936 9718 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee259c731e28486691317940ac6b211e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee259c731e28486691317940ac6b211e. Latest consensus state: current_term: 1 leader_uuid: "ee259c731e28486691317940ac6b211e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee259c731e28486691317940ac6b211e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38427 } } }
I20260926 16:46:22.772073 9718 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee259c731e28486691317940ac6b211e [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:22.771926 9717 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee259c731e28486691317940ac6b211e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee259c731e28486691317940ac6b211e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee259c731e28486691317940ac6b211e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38427 } } }
I20260926 16:46:22.772328 9717 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee259c731e28486691317940ac6b211e [sys.catalog]: This master's current role is: LEADER
W20260926 16:46:22.774803 9731 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ee259c731e28486691317940ac6b211e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 16:46:22.774865 9731 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 16:46:22.774945 9730 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:46:22.775581 9730 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:46:22.780905 9730 catalog_manager.cc:1347] Generated new cluster ID: bdb9ffee4b9c4648a3a15148885790b8
I20260926 16:46:22.780966 9730 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:46:22.822916 9730 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:46:22.823259 9730 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:46:22.832381 9730 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee259c731e28486691317940ac6b211e: Generated new TSK 0
I20260926 16:46:22.833143 9730 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:46:22.841079 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 9637
2026-09-26T16:46:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1418 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440924937351-21553-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:46:22 dist-test-slave-n1hw krb5kdc[9737](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:46:22 dist-test-slave-n1hw krb5kdc[9737](info): set up 2 sockets
Sep 26 16:46:22 dist-test-slave-n1hw krb5kdc[9737](info): commencing operation
krb5kdc: starting...
W20260926 16:46:23.900192 21553 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Sep 26 16:46:23 dist-test-slave-n1hw krb5kdc[9737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:46:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:46:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440924937351-21553-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440924937351-21553-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 16:46:24.020946 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36187
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440924937351-21553-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:46:24.128692 9750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:24.128996 9750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:24.129083 9750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:24.133060 9750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:46:24.133148 9750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:24.133186 9750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:24.133222 9750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:46:24.133253 9750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:46:24.138342 9750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440924937351-21553-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:46:24.139355 9750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:24.140370 9750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:24.148698 9756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:24.148847 9750 server_base.cc:1034] running on GCE node
W20260926 16:46:24.148720 9755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:24.148721 9758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:24.149530 9750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:24.150643 9750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:24.151813 9750 hybrid_clock.cc:648] HybridClock initialized: now 1790441184151801 us; error 24 us; skew 500 ppm
Sep 26 16:46:24 dist-test-slave-n1hw krb5kdc[9737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:46:24.159597 9750 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 16:46:24.160519 9750 webserver.cc:460] Webserver started at http://127.0.0.1:41197/ using document root <none> and password file <none>
I20260926 16:46:24.161072 9750 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:24.161163 9750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:24.161393 9750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:46:24.163477 9750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "5b2f18a500564d4a878d263587325f28"
format_stamp: "Formatted at 2026-09-26 16:46:24 on dist-test-slave-n1hw"
server_key: "288e30b70809df321b5f71c93e122ca0"
server_key_iv: "494cb6fae393bf4d4fe7f147c876a4d3"
server_key_version: "encryptionkey@0"
I20260926 16:46:24.164041 9750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "5b2f18a500564d4a878d263587325f28"
format_stamp: "Formatted at 2026-09-26 16:46:24 on dist-test-slave-n1hw"
server_key: "288e30b70809df321b5f71c93e122ca0"
server_key_iv: "494cb6fae393bf4d4fe7f147c876a4d3"
server_key_version: "encryptionkey@0"
I20260926 16:46:24.167645 9750 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260926 16:46:24.170045 9765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:24.171288 9750 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 16:46:24.171422 9750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "5b2f18a500564d4a878d263587325f28"
format_stamp: "Formatted at 2026-09-26 16:46:24 on dist-test-slave-n1hw"
server_key: "288e30b70809df321b5f71c93e122ca0"
server_key_iv: "494cb6fae393bf4d4fe7f147c876a4d3"
server_key_version: "encryptionkey@0"
I20260926 16:46:24.171530 9750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:24.206225 9750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:24.209299 9750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:24.209511 9750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:24.216761 9750 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36187
I20260926 16:46:24.216810 9817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36187 every 8 connection(s)
I20260926 16:46:24.217767 9750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:46:24.220544 9818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:24.226014 9818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b2f18a500564d4a878d263587325f28: Bootstrap starting.
I20260926 16:46:24.226872 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 9750
I20260926 16:46:24.227012 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:46:24.227284 21553 external_mini_cluster.cc:1428] Setting key 02a41a9d2223f51831755be31438068a
I20260926 16:46:24.228770 9818 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b2f18a500564d4a878d263587325f28: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:24.229494 9818 log.cc:824] T 00000000000000000000000000000000 P 5b2f18a500564d4a878d263587325f28: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:24.231400 9818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b2f18a500564d4a878d263587325f28: No bootstrap required, opened a new log
I20260926 16:46:24.234810 9818 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b2f18a500564d4a878d263587325f28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2f18a500564d4a878d263587325f28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36187 } }
I20260926 16:46:24.235045 9818 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b2f18a500564d4a878d263587325f28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:46:24.235131 9818 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b2f18a500564d4a878d263587325f28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b2f18a500564d4a878d263587325f28, State: Initialized, Role: FOLLOWER
I20260926 16:46:24.235641 9818 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b2f18a500564d4a878d263587325f28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2f18a500564d4a878d263587325f28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36187 } }
I20260926 16:46:24.235810 9818 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b2f18a500564d4a878d263587325f28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
Sep 26 16:46:24 dist-test-slave-n1hw krb5kdc[9737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 16:46:24.235874 9818 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b2f18a500564d4a878d263587325f28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:46:24.235975 9818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b2f18a500564d4a878d263587325f28 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:24.236977 9818 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b2f18a500564d4a878d263587325f28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2f18a500564d4a878d263587325f28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36187 } }
I20260926 16:46:24.237315 9818 leader_election.cc:302] T 00000000000000000000000000000000 P 5b2f18a500564d4a878d263587325f28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b2f18a500564d4a878d263587325f28; no voters:
I20260926 16:46:24.237597 9818 leader_election.cc:288] T 00000000000000000000000000000000 P 5b2f18a500564d4a878d263587325f28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:46:24.237717 9824 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b2f18a500564d4a878d263587325f28 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:46:24.237953 9824 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b2f18a500564d4a878d263587325f28 [term 1 LEADER]: Becoming Leader. State: Replica: 5b2f18a500564d4a878d263587325f28, State: Running, Role: LEADER
I20260926 16:46:24.238327 9824 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b2f18a500564d4a878d263587325f28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2f18a500564d4a878d263587325f28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36187 } }
I20260926 16:46:24.238880 9818 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b2f18a500564d4a878d263587325f28 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:46:24.239997 9825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b2f18a500564d4a878d263587325f28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b2f18a500564d4a878d263587325f28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2f18a500564d4a878d263587325f28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36187 } } }
I20260926 16:46:24.240371 9825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b2f18a500564d4a878d263587325f28 [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:24.240263 9826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b2f18a500564d4a878d263587325f28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b2f18a500564d4a878d263587325f28. Latest consensus state: current_term: 1 leader_uuid: "5b2f18a500564d4a878d263587325f28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2f18a500564d4a878d263587325f28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36187 } } }
I20260926 16:46:24.240705 9826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b2f18a500564d4a878d263587325f28 [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:24.241529 9836 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:46:24.242389 9821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:24.228459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59705 (local address 127.0.0.1:36187)
0926 16:46:24.228906 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:24.228921 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:24.228955 (+ 34us) server_negotiation.cc:407] Connection header received
0926 16:46:24.229573 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:24.229590 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:24.230053 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:24.230379 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:24.231180 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:24.232239 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:24.233095 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:24.233568 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:24.233746 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:24.236515 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:24.236520 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:24.236532 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:24.236558 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:24.238645 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:24.239143 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:24.239149 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:24.239154 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:24.239294 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:24.239640 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:24.239645 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:24.239646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:24.240198 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:24.240403 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:24.241309 (+ 906us) server_negotiation.cc:299] Negotiation successful
0926 16:46:24.241533 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":140,"threads_started":1}
I20260926 16:46:24.244994 9836 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:46:24.245481 21553 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:46:24.250540 9836 catalog_manager.cc:1347] Generated new cluster ID: 8bdf256ebfbf4a0db61770f8b96d111b
I20260926 16:46:24.250599 9836 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:46:24.256675 9821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:24.247754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47753 (local address 127.0.0.1:36187)
0926 16:46:24.247851 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:24.247855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:24.248637 (+ 782us) server_negotiation.cc:407] Connection header received
0926 16:46:24.248735 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:24.248738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:24.248784 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:24.248868 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:24.249559 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:24.250144 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:24.251276 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:24.251647 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:24.251756 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:24.253247 (+ 1491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:24.253251 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:24.253255 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:24.253279 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:24.255032 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:24.255488 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:24.255492 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:24.255494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:24.255584 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:24.255865 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:24.255868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:24.255870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:24.256069 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:24.256207 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:24.256405 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 16:46:24.256515 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 16:46:24.269699 9836 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:46:24.270696 9836 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:46:24.288632 9836 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b2f18a500564d4a878d263587325f28: Generated new TSK 0
I20260926 16:46:24.289484 9836 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:46:24.294488 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 9750
2026-09-26T16:46:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1452 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440924937351-21553-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:46:24 dist-test-slave-n1hw krb5kdc[9851](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:46:24 dist-test-slave-n1hw krb5kdc[9851](info): set up 2 sockets
Sep 26 16:46:24 dist-test-slave-n1hw krb5kdc[9851](info): commencing operation
krb5kdc: starting...
W20260926 16:46:25.340281 21553 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Sep 26 16:46:25 dist-test-slave-n1hw krb5kdc[9851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:46:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:46:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440924937351-21553-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440924937351-21553-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 16:46:25.462904 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34805
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440924937351-21553-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:46:25.573271 9864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:25.573544 9864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:25.573587 9864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:25.577601 9864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:46:25.577656 9864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:25.577675 9864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:25.577692 9864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:46:25.577708 9864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:46:25.582898 9864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440924937351-21553-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:46:25.583918 9864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:25.584919 9864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:25.593384 9869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:25.593416 9872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:25.593478 9864 server_base.cc:1034] running on GCE node
W20260926 16:46:25.593384 9870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:25.594254 9864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:25.595376 9864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:25.596580 9864 hybrid_clock.cc:648] HybridClock initialized: now 1790441185596543 us; error 46 us; skew 500 ppm
Sep 26 16:46:25 dist-test-slave-n1hw krb5kdc[9851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:46:25.599567 9864 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 16:46:25.600510 9864 webserver.cc:460] Webserver started at http://127.0.0.1:46413/ using document root <none> and password file <none>
I20260926 16:46:25.601070 9864 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:25.601158 9864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:25.601389 9864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:46:25.603451 9864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "436af7b2cda3428284657b32516eef0d"
format_stamp: "Formatted at 2026-09-26 16:46:25 on dist-test-slave-n1hw"
server_key: "35bac10573571aa2931223aefaaf8c4a"
server_key_iv: "506ae9752bbc699d4c4747eb4584d5d7"
server_key_version: "encryptionkey@0"
I20260926 16:46:25.604012 9864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "436af7b2cda3428284657b32516eef0d"
format_stamp: "Formatted at 2026-09-26 16:46:25 on dist-test-slave-n1hw"
server_key: "35bac10573571aa2931223aefaaf8c4a"
server_key_iv: "506ae9752bbc699d4c4747eb4584d5d7"
server_key_version: "encryptionkey@0"
I20260926 16:46:25.607640 9864 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260926 16:46:25.610031 9879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:25.611284 9864 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:46:25.611431 9864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "436af7b2cda3428284657b32516eef0d"
format_stamp: "Formatted at 2026-09-26 16:46:25 on dist-test-slave-n1hw"
server_key: "35bac10573571aa2931223aefaaf8c4a"
server_key_iv: "506ae9752bbc699d4c4747eb4584d5d7"
server_key_version: "encryptionkey@0"
I20260926 16:46:25.611542 9864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:25.623301 9864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:25.626536 9864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:25.626725 9864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:25.633868 9864 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34805
I20260926 16:46:25.633894 9931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34805 every 8 connection(s)
I20260926 16:46:25.634763 9864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:46:25.637445 9932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:25.638811 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 9864
I20260926 16:46:25.638949 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:46:25.639256 21553 external_mini_cluster.cc:1428] Setting key 1f90eb2f597d3088b9380984d085a660
I20260926 16:46:25.643371 9932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 436af7b2cda3428284657b32516eef0d: Bootstrap starting.
I20260926 16:46:25.645460 9932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 436af7b2cda3428284657b32516eef0d: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:25.646318 9932 log.cc:824] T 00000000000000000000000000000000 P 436af7b2cda3428284657b32516eef0d: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:25.648236 9935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:25.640611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59097 (local address 127.0.0.1:34805)
0926 16:46:25.641035 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:25.641049 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:25.641086 (+ 37us) server_negotiation.cc:407] Connection header received
0926 16:46:25.641702 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:25.641718 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:25.642105 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:25.642415 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:25.642851 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:25.643818 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:25.644685 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:25.645150 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:25.645331 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:25.646291 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:25.646298 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:25.646310 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:46:25.646844 (+ 534us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:25.646914 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:46:25.646966 (+ 52us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:25.647099 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:25.647208 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 16:46:25.647446 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":129,"threads_started":1}
I20260926 16:46:25.648456 9932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 436af7b2cda3428284657b32516eef0d: No bootstrap required, opened a new log
I20260926 16:46:25.651134 21553 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:46:25.652299 9932 raft_consensus.cc:348] T 00000000000000000000000000000000 P 436af7b2cda3428284657b32516eef0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "436af7b2cda3428284657b32516eef0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34805 } }
I20260926 16:46:25.652595 9932 raft_consensus.cc:374] T 00000000000000000000000000000000 P 436af7b2cda3428284657b32516eef0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:46:25.652657 9932 raft_consensus.cc:729] T 00000000000000000000000000000000 P 436af7b2cda3428284657b32516eef0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 436af7b2cda3428284657b32516eef0d, State: Initialized, Role: FOLLOWER
I20260926 16:46:25.653108 9932 consensus_queue.cc:260] T 00000000000000000000000000000000 P 436af7b2cda3428284657b32516eef0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "436af7b2cda3428284657b32516eef0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34805 } }
I20260926 16:46:25.653282 9932 raft_consensus.cc:388] T 00000000000000000000000000000000 P 436af7b2cda3428284657b32516eef0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:46:25.653379 9932 raft_consensus.cc:482] T 00000000000000000000000000000000 P 436af7b2cda3428284657b32516eef0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:46:25.653487 9932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 436af7b2cda3428284657b32516eef0d [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:25.654596 9932 raft_consensus.cc:504] T 00000000000000000000000000000000 P 436af7b2cda3428284657b32516eef0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "436af7b2cda3428284657b32516eef0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34805 } }
I20260926 16:46:25.654960 9932 leader_election.cc:302] T 00000000000000000000000000000000 P 436af7b2cda3428284657b32516eef0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 436af7b2cda3428284657b32516eef0d; no voters:
I20260926 16:46:25.655283 9932 leader_election.cc:288] T 00000000000000000000000000000000 P 436af7b2cda3428284657b32516eef0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:46:25.655365 9942 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 436af7b2cda3428284657b32516eef0d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:46:25.655647 9942 raft_consensus.cc:686] T 00000000000000000000000000000000 P 436af7b2cda3428284657b32516eef0d [term 1 LEADER]: Becoming Leader. State: Replica: 436af7b2cda3428284657b32516eef0d, State: Running, Role: LEADER
I20260926 16:46:25.656002 9942 consensus_queue.cc:237] T 00000000000000000000000000000000 P 436af7b2cda3428284657b32516eef0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "436af7b2cda3428284657b32516eef0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34805 } }
I20260926 16:46:25.656474 9932 sys_catalog.cc:564] T 00000000000000000000000000000000 P 436af7b2cda3428284657b32516eef0d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:46:25.657706 9944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 436af7b2cda3428284657b32516eef0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 436af7b2cda3428284657b32516eef0d. Latest consensus state: current_term: 1 leader_uuid: "436af7b2cda3428284657b32516eef0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "436af7b2cda3428284657b32516eef0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34805 } } }
I20260926 16:46:25.657928 9943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 436af7b2cda3428284657b32516eef0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "436af7b2cda3428284657b32516eef0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "436af7b2cda3428284657b32516eef0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34805 } } }
I20260926 16:46:25.658066 9943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 436af7b2cda3428284657b32516eef0d [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:25.658249 9935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:25.653331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43515 (local address 127.0.0.1:34805)
0926 16:46:25.653456 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:25.653460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:25.653717 (+ 257us) server_negotiation.cc:407] Connection header received
0926 16:46:25.653862 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:25.653864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:25.653900 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:25.653978 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:25.654277 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:25.654849 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:25.656001 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:25.656326 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:25.656430 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:25.657363 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:25.657369 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:25.657372 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:46:25.657385 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:25.657418 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:46:25.657433 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:25.657511 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:25.658056 (+ 545us) server_negotiation.cc:299] Negotiation successful
0926 16:46:25.658112 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 16:46:25.658489 9944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 436af7b2cda3428284657b32516eef0d [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:25.659442 9953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:46:25.662127 9953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:46:25.666965 9953 catalog_manager.cc:1347] Generated new cluster ID: afa23f3ec0b04fc18967f9fa0a213f46
I20260926 16:46:25.667084 9953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:46:25.690136 9953 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:46:25.691222 9953 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:46:25.699250 9953 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 436af7b2cda3428284657b32516eef0d: Generated new TSK 0
I20260926 16:46:25.700090 9953 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:46:25.704552 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 9864
2026-09-26T16:46:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1411 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440924937351-21553-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:46:25 dist-test-slave-n1hw krb5kdc[9964](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:46:25 dist-test-slave-n1hw krb5kdc[9964](info): set up 2 sockets
Sep 26 16:46:25 dist-test-slave-n1hw krb5kdc[9964](info): commencing operation
krb5kdc: starting...
W20260926 16:46:26.776010 21553 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Sep 26 16:46:26 dist-test-slave-n1hw krb5kdc[9964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:46:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:46:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440924937351-21553-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440924937351-21553-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 16:46:26.900840 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36427
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440924937351-21553-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36427
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:46:27.010135 9977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:27.010387 9977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:27.010427 9977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:27.014464 9977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:46:27.014516 9977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:27.014535 9977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:27.014551 9977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:46:27.014604 9977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:46:27.019644 9977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440924937351-21553-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:46:27.020617 9977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:27.021617 9977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:27.029534 9985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:27.029501 9982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:27.029524 9983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:27.029654 9977 server_base.cc:1034] running on GCE node
I20260926 16:46:27.030350 9977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:27.031368 9977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:27.032589 9977 hybrid_clock.cc:648] HybridClock initialized: now 1790441187032545 us; error 30 us; skew 500 ppm
Sep 26 16:46:27 dist-test-slave-n1hw krb5kdc[9964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:46:27.035369 9977 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 16:46:27.036237 9977 webserver.cc:460] Webserver started at http://127.0.0.1:43543/ using document root <none> and password file <none>
I20260926 16:46:27.036778 9977 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:27.036867 9977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:27.037092 9977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:46:27.039101 9977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "89f6698569864ac78750a22e3208d142"
format_stamp: "Formatted at 2026-09-26 16:46:27 on dist-test-slave-n1hw"
server_key: "3b3d2743c1ca94771696127fa47305b8"
server_key_iv: "b923fa348b8680c3b0a0f73e37f6dedf"
server_key_version: "encryptionkey@0"
I20260926 16:46:27.039655 9977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "89f6698569864ac78750a22e3208d142"
format_stamp: "Formatted at 2026-09-26 16:46:27 on dist-test-slave-n1hw"
server_key: "3b3d2743c1ca94771696127fa47305b8"
server_key_iv: "b923fa348b8680c3b0a0f73e37f6dedf"
server_key_version: "encryptionkey@0"
I20260926 16:46:27.043174 9977 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260926 16:46:27.045450 9992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:27.046689 9977 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:46:27.046823 9977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "89f6698569864ac78750a22e3208d142"
format_stamp: "Formatted at 2026-09-26 16:46:27 on dist-test-slave-n1hw"
server_key: "3b3d2743c1ca94771696127fa47305b8"
server_key_iv: "b923fa348b8680c3b0a0f73e37f6dedf"
server_key_version: "encryptionkey@0"
I20260926 16:46:27.046924 9977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:27.051349 9977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:27.054652 9977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:27.054847 9977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:27.061443 9977 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36427
I20260926 16:46:27.061510 10044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36427 every 8 connection(s)
I20260926 16:46:27.062424 9977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:46:27.065136 10045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:27.066956 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 9977
I20260926 16:46:27.067117 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:46:27.067412 21553 external_mini_cluster.cc:1428] Setting key 11170d69ebe0be5d3cbc38558e592f92
I20260926 16:46:27.071550 10045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89f6698569864ac78750a22e3208d142: Bootstrap starting.
I20260926 16:46:27.072501 10048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:27.068728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51211 (local address 127.0.0.1:36427)
0926 16:46:27.069168 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:27.069183 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:27.069216 (+ 33us) server_negotiation.cc:407] Connection header received
0926 16:46:27.069924 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:27.069943 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:27.070224 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:27.070563 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:27.071378 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:27.071382 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:27.071394 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:46:27.071405 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:27.071459 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:46:27.071478 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:27.071626 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:27.071698 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 16:46:27.071844 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":140,"threads_started":1}
I20260926 16:46:27.074013 10045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89f6698569864ac78750a22e3208d142: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:27.074855 10045 log.cc:824] T 00000000000000000000000000000000 P 89f6698569864ac78750a22e3208d142: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:27.075356 21553 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:46:27.077149 10045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89f6698569864ac78750a22e3208d142: No bootstrap required, opened a new log
I20260926 16:46:27.079452 10048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:27.077593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34061 (local address 127.0.0.1:36427)
0926 16:46:27.077711 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:27.077714 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:27.078017 (+ 303us) server_negotiation.cc:407] Connection header received
0926 16:46:27.078168 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:27.078171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:27.078202 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:27.078294 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:27.079020 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:27.079026 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:27.079037 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:46:27.079048 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:27.079075 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:46:27.079094 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:27.079181 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:27.079295 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 16:46:27.079336 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 16:46:27.081226 10045 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89f6698569864ac78750a22e3208d142 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89f6698569864ac78750a22e3208d142" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36427 } }
I20260926 16:46:27.081439 10045 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89f6698569864ac78750a22e3208d142 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:46:27.081512 10045 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89f6698569864ac78750a22e3208d142 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f6698569864ac78750a22e3208d142, State: Initialized, Role: FOLLOWER
I20260926 16:46:27.081959 10045 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89f6698569864ac78750a22e3208d142 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89f6698569864ac78750a22e3208d142" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36427 } }
I20260926 16:46:27.082082 10045 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89f6698569864ac78750a22e3208d142 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:46:27.082123 10045 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89f6698569864ac78750a22e3208d142 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:46:27.082185 10045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89f6698569864ac78750a22e3208d142 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:27.083151 10045 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89f6698569864ac78750a22e3208d142 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89f6698569864ac78750a22e3208d142" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36427 } }
I20260926 16:46:27.083513 10045 leader_election.cc:302] T 00000000000000000000000000000000 P 89f6698569864ac78750a22e3208d142 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89f6698569864ac78750a22e3208d142; no voters:
I20260926 16:46:27.083820 10045 leader_election.cc:288] T 00000000000000000000000000000000 P 89f6698569864ac78750a22e3208d142 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:46:27.083906 10055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89f6698569864ac78750a22e3208d142 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:46:27.084183 10055 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89f6698569864ac78750a22e3208d142 [term 1 LEADER]: Becoming Leader. State: Replica: 89f6698569864ac78750a22e3208d142, State: Running, Role: LEADER
I20260926 16:46:27.084528 10055 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89f6698569864ac78750a22e3208d142 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89f6698569864ac78750a22e3208d142" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36427 } }
I20260926 16:46:27.085057 10045 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89f6698569864ac78750a22e3208d142 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:46:27.086200 10057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89f6698569864ac78750a22e3208d142 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89f6698569864ac78750a22e3208d142. Latest consensus state: current_term: 1 leader_uuid: "89f6698569864ac78750a22e3208d142" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89f6698569864ac78750a22e3208d142" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36427 } } }
I20260926 16:46:27.086340 10057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89f6698569864ac78750a22e3208d142 [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:27.086730 10064 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:46:27.086688 10056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89f6698569864ac78750a22e3208d142 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89f6698569864ac78750a22e3208d142" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89f6698569864ac78750a22e3208d142" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36427 } } }
I20260926 16:46:27.087313 10056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89f6698569864ac78750a22e3208d142 [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:27.089864 10064 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:46:27.095436 10064 catalog_manager.cc:1347] Generated new cluster ID: d4eff823f7f4427b84454bfe50c647ae
I20260926 16:46:27.095502 10064 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:46:27.105057 10064 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:46:27.105443 10064 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:46:27.111104 10064 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89f6698569864ac78750a22e3208d142: Generated new TSK 0
I20260926 16:46:27.111756 10064 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:46:27.115917 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 9977
2026-09-26T16:46:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1409 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440924937351-21553-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:46:27 dist-test-slave-n1hw krb5kdc[10076](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:46:27 dist-test-slave-n1hw krb5kdc[10076](info): set up 2 sockets
Sep 26 16:46:27 dist-test-slave-n1hw krb5kdc[10076](info): commencing operation
krb5kdc: starting...
W20260926 16:46:28.160032 21553 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Sep 26 16:46:28 dist-test-slave-n1hw krb5kdc[10076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:46:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:46:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440924937351-21553-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440924937351-21553-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 16:46:28.279994 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37367
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440924937351-21553-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:46:28.395998 10090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:28.396289 10090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:28.396382 10090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:28.400364 10090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:46:28.400453 10090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:28.400493 10090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:28.400552 10090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:46:28.400592 10090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:46:28.405723 10090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440924937351-21553-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:46:28.406787 10090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:28.407796 10090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:28.416007 10095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:28.416242 10090 server_base.cc:1034] running on GCE node
W20260926 16:46:28.416055 10096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:28.416002 10098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:28.417009 10090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:28.418195 10090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:28.419361 10090 hybrid_clock.cc:648] HybridClock initialized: now 1790441188419348 us; error 27 us; skew 500 ppm
Sep 26 16:46:28 dist-test-slave-n1hw krb5kdc[10076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:46:28.422312 10090 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 16:46:28.423234 10090 webserver.cc:460] Webserver started at http://127.0.0.1:34799/ using document root <none> and password file <none>
I20260926 16:46:28.423803 10090 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:28.423861 10090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:28.424064 10090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:46:28.426101 10090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "943253cc16b34713b0e45dce0cbd3540"
format_stamp: "Formatted at 2026-09-26 16:46:28 on dist-test-slave-n1hw"
server_key: "85499867dcd1a42d860b770542d4239b"
server_key_iv: "b1c2c18bf9d14923a28764402367392e"
server_key_version: "encryptionkey@0"
I20260926 16:46:28.426661 10090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "943253cc16b34713b0e45dce0cbd3540"
format_stamp: "Formatted at 2026-09-26 16:46:28 on dist-test-slave-n1hw"
server_key: "85499867dcd1a42d860b770542d4239b"
server_key_iv: "b1c2c18bf9d14923a28764402367392e"
server_key_version: "encryptionkey@0"
I20260926 16:46:28.430217 10090 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:46:28.432526 10105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:28.433691 10090 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:46:28.433987 10090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "943253cc16b34713b0e45dce0cbd3540"
format_stamp: "Formatted at 2026-09-26 16:46:28 on dist-test-slave-n1hw"
server_key: "85499867dcd1a42d860b770542d4239b"
server_key_iv: "b1c2c18bf9d14923a28764402367392e"
server_key_version: "encryptionkey@0"
I20260926 16:46:28.434091 10090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:28.445627 10090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:28.448800 10090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:28.448997 10090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:28.456230 10090 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37367
I20260926 16:46:28.456249 10157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37367 every 8 connection(s)
I20260926 16:46:28.457140 10090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:46:28.459954 10158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:28.465425 10158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 943253cc16b34713b0e45dce0cbd3540: Bootstrap starting.
I20260926 16:46:28.465905 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 10090
I20260926 16:46:28.466056 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:46:28.466380 21553 external_mini_cluster.cc:1428] Setting key af63b24df6fb8e07ac215d2f68fe09b1
I20260926 16:46:28.468315 10158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 943253cc16b34713b0e45dce0cbd3540: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:28.469053 10158 log.cc:824] T 00000000000000000000000000000000 P 943253cc16b34713b0e45dce0cbd3540: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:28.471230 10158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 943253cc16b34713b0e45dce0cbd3540: No bootstrap required, opened a new log
Sep 26 16:46:28 dist-test-slave-n1hw krb5kdc[10076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 16:46:28.475711 10158 raft_consensus.cc:348] T 00000000000000000000000000000000 P 943253cc16b34713b0e45dce0cbd3540 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "943253cc16b34713b0e45dce0cbd3540" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37367 } }
I20260926 16:46:28.475926 10158 raft_consensus.cc:374] T 00000000000000000000000000000000 P 943253cc16b34713b0e45dce0cbd3540 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:46:28.476016 10158 raft_consensus.cc:729] T 00000000000000000000000000000000 P 943253cc16b34713b0e45dce0cbd3540 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 943253cc16b34713b0e45dce0cbd3540, State: Initialized, Role: FOLLOWER
I20260926 16:46:28.476449 10158 consensus_queue.cc:260] T 00000000000000000000000000000000 P 943253cc16b34713b0e45dce0cbd3540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "943253cc16b34713b0e45dce0cbd3540" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37367 } }
I20260926 16:46:28.476603 10158 raft_consensus.cc:388] T 00000000000000000000000000000000 P 943253cc16b34713b0e45dce0cbd3540 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:46:28.476689 10158 raft_consensus.cc:482] T 00000000000000000000000000000000 P 943253cc16b34713b0e45dce0cbd3540 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:46:28.476820 10158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 943253cc16b34713b0e45dce0cbd3540 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:28.477959 10158 raft_consensus.cc:504] T 00000000000000000000000000000000 P 943253cc16b34713b0e45dce0cbd3540 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "943253cc16b34713b0e45dce0cbd3540" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37367 } }
I20260926 16:46:28.478304 10158 leader_election.cc:302] T 00000000000000000000000000000000 P 943253cc16b34713b0e45dce0cbd3540 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 943253cc16b34713b0e45dce0cbd3540; no voters:
I20260926 16:46:28.478645 10158 leader_election.cc:288] T 00000000000000000000000000000000 P 943253cc16b34713b0e45dce0cbd3540 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:46:28.478734 10163 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 943253cc16b34713b0e45dce0cbd3540 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:46:28.479027 10163 raft_consensus.cc:686] T 00000000000000000000000000000000 P 943253cc16b34713b0e45dce0cbd3540 [term 1 LEADER]: Becoming Leader. State: Replica: 943253cc16b34713b0e45dce0cbd3540, State: Running, Role: LEADER
I20260926 16:46:28.479368 10163 consensus_queue.cc:237] T 00000000000000000000000000000000 P 943253cc16b34713b0e45dce0cbd3540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "943253cc16b34713b0e45dce0cbd3540" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37367 } }
I20260926 16:46:28.480041 10158 sys_catalog.cc:564] T 00000000000000000000000000000000 P 943253cc16b34713b0e45dce0cbd3540 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:46:28.480396 10163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 943253cc16b34713b0e45dce0cbd3540 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 943253cc16b34713b0e45dce0cbd3540. Latest consensus state: current_term: 1 leader_uuid: "943253cc16b34713b0e45dce0cbd3540" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "943253cc16b34713b0e45dce0cbd3540" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37367 } } }
I20260926 16:46:28.480578 10163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 943253cc16b34713b0e45dce0cbd3540 [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:28.480851 10164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 943253cc16b34713b0e45dce0cbd3540 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "943253cc16b34713b0e45dce0cbd3540" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "943253cc16b34713b0e45dce0cbd3540" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37367 } } }
I20260926 16:46:28.480986 10164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 943253cc16b34713b0e45dce0cbd3540 [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:28.481003 10167 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:46:28.482710 10161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:28.467798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:56843 (local address 127.0.0.1:37367)
0926 16:46:28.468217 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:28.468231 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:28.468265 (+ 34us) server_negotiation.cc:407] Connection header received
0926 16:46:28.468927 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:28.468944 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:28.469312 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:28.469675 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:28.470705 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:28.471739 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:28.472962 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:28.473433 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:28.473548 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:28.476574 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:28.476579 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:28.476611 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:28.476681 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:28.478540 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:28.479174 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:28.479180 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:28.479186 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:28.479315 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:28.479718 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:28.479721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:28.479723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:28.480207 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:28.481151 (+ 944us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:28.481333 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 16:46:28.481572 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":131,"threads_started":1}
I20260926 16:46:28.484161 10167 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:46:28.486224 21553 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:46:28.489854 10167 catalog_manager.cc:1347] Generated new cluster ID: 898dbb8b28d648c3ab646e6bc0a319ff
I20260926 16:46:28.489907 10167 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:46:28.497319 10161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:28.488284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:54911 (local address 127.0.0.1:37367)
0926 16:46:28.488394 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:28.488398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:28.488607 (+ 209us) server_negotiation.cc:407] Connection header received
0926 16:46:28.488753 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:28.488757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:28.488809 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:28.488858 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:28.489784 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:28.490657 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:28.491859 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:28.492209 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:28.492302 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:28.493932 (+ 1630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:28.493936 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:28.493938 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:28.493961 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:28.495573 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:28.496087 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:28.496091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:28.496093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:28.496176 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:28.496505 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:28.496508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:28.496509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:28.496688 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:28.496831 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:28.497088 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 16:46:28.497181 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 16:46:28.503108 10167 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:46:28.504237 10167 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:46:28.510458 10167 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 943253cc16b34713b0e45dce0cbd3540: Generated new TSK 0
I20260926 16:46:28.511307 10167 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:46:28.515784 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 10090
W20260926 16:46:28.521088 10074 connection.cc:535] client connection to 127.0.0.1:37367 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37367): Connection reset by peer (error 104)
2026-09-26T16:46:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1400 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440924937351-21553-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:46:28 dist-test-slave-n1hw krb5kdc[10189](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:46:28 dist-test-slave-n1hw krb5kdc[10189](info): set up 2 sockets
Sep 26 16:46:28 dist-test-slave-n1hw krb5kdc[10189](info): commencing operation
krb5kdc: starting...
W20260926 16:46:29.560194 21553 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Sep 26 16:46:29 dist-test-slave-n1hw krb5kdc[10189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:46:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:46:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440924937351-21553-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440924937351-21553-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 16:46:29.681501 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46593
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440924937351-21553-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:46:29.792445 10202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:29.792699 10202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:29.792742 10202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:29.796900 10202 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:46:29.796958 10202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:29.796977 10202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:29.796994 10202 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:46:29.797050 10202 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:46:29.802353 10202 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440924937351-21553-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:46:29.803817 10202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:29.804843 10202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:29.813097 10210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:29.813098 10208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:29.813133 10207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:29.813367 10202 server_base.cc:1034] running on GCE node
I20260926 16:46:29.813982 10202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:29.815069 10202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:29.816270 10202 hybrid_clock.cc:648] HybridClock initialized: now 1790441189816246 us; error 42 us; skew 500 ppm
Sep 26 16:46:29 dist-test-slave-n1hw krb5kdc[10189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:46:29.823323 10202 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 16:46:29.824327 10202 webserver.cc:460] Webserver started at http://127.0.0.1:36823/ using document root <none> and password file <none>
I20260926 16:46:29.824901 10202 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:29.824993 10202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:29.825241 10202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:46:29.827464 10202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "328733af6ab7424993b7d2d283189d30"
format_stamp: "Formatted at 2026-09-26 16:46:29 on dist-test-slave-n1hw"
server_key: "80fb3a5b0fafcdae84b5bd0640654259"
server_key_iv: "c6d17ecc825721c23e0c467c7ea45f30"
server_key_version: "encryptionkey@0"
I20260926 16:46:29.828038 10202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "328733af6ab7424993b7d2d283189d30"
format_stamp: "Formatted at 2026-09-26 16:46:29 on dist-test-slave-n1hw"
server_key: "80fb3a5b0fafcdae84b5bd0640654259"
server_key_iv: "c6d17ecc825721c23e0c467c7ea45f30"
server_key_version: "encryptionkey@0"
I20260926 16:46:29.831950 10202 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260926 16:46:29.834558 10217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:29.835961 10202 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:46:29.836089 10202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "328733af6ab7424993b7d2d283189d30"
format_stamp: "Formatted at 2026-09-26 16:46:29 on dist-test-slave-n1hw"
server_key: "80fb3a5b0fafcdae84b5bd0640654259"
server_key_iv: "c6d17ecc825721c23e0c467c7ea45f30"
server_key_version: "encryptionkey@0"
I20260926 16:46:29.836200 10202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:29.846949 10202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:29.849897 10202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:29.850096 10202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:29.857793 10202 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46593
I20260926 16:46:29.857877 10270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46593 every 8 connection(s)
I20260926 16:46:29.858852 10202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:46:29.861611 10271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:29.866923 10271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 328733af6ab7424993b7d2d283189d30: Bootstrap starting.
I20260926 16:46:29.867514 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 10202
I20260926 16:46:29.867654 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:46:29.867947 21553 external_mini_cluster.cc:1428] Setting key aad110712585e784ae9f972c6a4f6873
I20260926 16:46:29.869709 10271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 328733af6ab7424993b7d2d283189d30: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:29.870437 10271 log.cc:824] T 00000000000000000000000000000000 P 328733af6ab7424993b7d2d283189d30: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:29.872622 10271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 328733af6ab7424993b7d2d283189d30: No bootstrap required, opened a new log
I20260926 16:46:29.876935 10271 raft_consensus.cc:348] T 00000000000000000000000000000000 P 328733af6ab7424993b7d2d283189d30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "328733af6ab7424993b7d2d283189d30" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46593 } }
I20260926 16:46:29.877184 10271 raft_consensus.cc:374] T 00000000000000000000000000000000 P 328733af6ab7424993b7d2d283189d30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:46:29.877233 10271 raft_consensus.cc:729] T 00000000000000000000000000000000 P 328733af6ab7424993b7d2d283189d30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 328733af6ab7424993b7d2d283189d30, State: Initialized, Role: FOLLOWER
I20260926 16:46:29.877632 10274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:29.869246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:58617 (local address 127.0.0.1:46593)
0926 16:46:29.869637 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:29.869652 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:29.869687 (+ 35us) server_negotiation.cc:407] Connection header received
0926 16:46:29.870364 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:29.870381 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:29.870761 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:29.871031 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:29.871456 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:29.872856 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:29.873785 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:29.874543 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:29.874699 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:29.875588 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:29.875595 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:29.875612 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:46:29.876101 (+ 489us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:29.876174 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:46:29.876214 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:29.876345 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:29.876557 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 16:46:29.876696 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":127,"threads_started":1}
I20260926 16:46:29.877689 10271 consensus_queue.cc:260] T 00000000000000000000000000000000 P 328733af6ab7424993b7d2d283189d30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "328733af6ab7424993b7d2d283189d30" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46593 } }
I20260926 16:46:29.877861 10271 raft_consensus.cc:388] T 00000000000000000000000000000000 P 328733af6ab7424993b7d2d283189d30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:46:29.877930 10271 raft_consensus.cc:482] T 00000000000000000000000000000000 P 328733af6ab7424993b7d2d283189d30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:46:29.878026 10271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 328733af6ab7424993b7d2d283189d30 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:29.879057 10271 raft_consensus.cc:504] T 00000000000000000000000000000000 P 328733af6ab7424993b7d2d283189d30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "328733af6ab7424993b7d2d283189d30" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46593 } }
I20260926 16:46:29.879408 10271 leader_election.cc:302] T 00000000000000000000000000000000 P 328733af6ab7424993b7d2d283189d30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 328733af6ab7424993b7d2d283189d30; no voters:
I20260926 16:46:29.879751 10271 leader_election.cc:288] T 00000000000000000000000000000000 P 328733af6ab7424993b7d2d283189d30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:46:29.879832 10276 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 328733af6ab7424993b7d2d283189d30 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:46:29.880077 10276 raft_consensus.cc:686] T 00000000000000000000000000000000 P 328733af6ab7424993b7d2d283189d30 [term 1 LEADER]: Becoming Leader. State: Replica: 328733af6ab7424993b7d2d283189d30, State: Running, Role: LEADER
I20260926 16:46:29.880661 21553 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:46:29.880745 10276 consensus_queue.cc:237] T 00000000000000000000000000000000 P 328733af6ab7424993b7d2d283189d30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "328733af6ab7424993b7d2d283189d30" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46593 } }
I20260926 16:46:29.881042 10271 sys_catalog.cc:564] T 00000000000000000000000000000000 P 328733af6ab7424993b7d2d283189d30 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:46:29.881911 10276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 328733af6ab7424993b7d2d283189d30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 328733af6ab7424993b7d2d283189d30. Latest consensus state: current_term: 1 leader_uuid: "328733af6ab7424993b7d2d283189d30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "328733af6ab7424993b7d2d283189d30" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46593 } } }
I20260926 16:46:29.882635 10276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 328733af6ab7424993b7d2d283189d30 [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:29.882524 10277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 328733af6ab7424993b7d2d283189d30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "328733af6ab7424993b7d2d283189d30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "328733af6ab7424993b7d2d283189d30" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46593 } } }
I20260926 16:46:29.882944 10277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 328733af6ab7424993b7d2d283189d30 [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:29.883774 10294 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:46:29.886852 10294 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:46:29.887532 10274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:29.882932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:49271 (local address 127.0.0.1:46593)
0926 16:46:29.883035 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:29.883040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:29.884005 (+ 965us) server_negotiation.cc:407] Connection header received
0926 16:46:29.884120 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:29.884122 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:29.884158 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:29.884216 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:29.884502 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:29.884991 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:29.885872 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:29.886117 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:29.886227 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:29.887083 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:29.887086 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:29.887089 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:46:29.887102 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:29.887133 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:46:29.887143 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:29.887237 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:29.887358 (+ 121us) server_negotiation.cc:299] Negotiation successful
0926 16:46:29.887389 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 16:46:29.892851 10294 catalog_manager.cc:1347] Generated new cluster ID: 5f733b01554540648e7347986333696e
I20260926 16:46:29.892930 10294 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:46:29.899644 10294 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:46:29.900800 10294 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:46:29.908581 10294 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 328733af6ab7424993b7d2d283189d30: Generated new TSK 0
I20260926 16:46:29.909338 10294 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:46:29.917068 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 10202
W20260926 16:46:29.922408 10187 connection.cc:535] client connection to 127.0.0.1:46593 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46593): Connection reset by peer (error 104)
2026-09-26T16:46:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1401 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440924937351-21553-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:46:29 dist-test-slave-n1hw krb5kdc[10302](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:46:29 dist-test-slave-n1hw krb5kdc[10302](info): set up 2 sockets
Sep 26 16:46:29 dist-test-slave-n1hw krb5kdc[10302](info): commencing operation
krb5kdc: starting...
W20260926 16:46:30.960492 21553 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Sep 26 16:46:31 dist-test-slave-n1hw krb5kdc[10302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:46:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:46:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440924937351-21553-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440924937351-21553-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 16:46:31.085392 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35941
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440924937351-21553-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35941
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:46:31.193674 10315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:31.194005 10315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:31.194046 10315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:31.197969 10315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:46:31.198025 10315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:31.198045 10315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:31.198060 10315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:46:31.198076 10315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:46:31.203228 10315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440924937351-21553-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:46:31.204206 10315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:31.205148 10315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:31.213620 10323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:31.213660 10321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:31.213616 10320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:31.213970 10315 server_base.cc:1034] running on GCE node
I20260926 16:46:31.214618 10315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:31.215703 10315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:31.216897 10315 hybrid_clock.cc:648] HybridClock initialized: now 1790441191216883 us; error 25 us; skew 500 ppm
Sep 26 16:46:31 dist-test-slave-n1hw krb5kdc[10302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:46:31.219599 10315 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 16:46:31.220410 10315 webserver.cc:460] Webserver started at http://127.0.0.1:42139/ using document root <none> and password file <none>
I20260926 16:46:31.220917 10315 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:31.220973 10315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:31.221127 10315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:46:31.223083 10315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "23e6ad8e198c4aad99ac6d3660f49612"
format_stamp: "Formatted at 2026-09-26 16:46:31 on dist-test-slave-n1hw"
server_key: "1dab491dcc51c32f2d334ee9eec089dd"
server_key_iv: "3eedf5b8f71ffb2d8dd5171362764f17"
server_key_version: "encryptionkey@0"
I20260926 16:46:31.223584 10315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "23e6ad8e198c4aad99ac6d3660f49612"
format_stamp: "Formatted at 2026-09-26 16:46:31 on dist-test-slave-n1hw"
server_key: "1dab491dcc51c32f2d334ee9eec089dd"
server_key_iv: "3eedf5b8f71ffb2d8dd5171362764f17"
server_key_version: "encryptionkey@0"
I20260926 16:46:31.227087 10315 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:46:31.229341 10330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:31.230624 10315 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:46:31.230720 10315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "23e6ad8e198c4aad99ac6d3660f49612"
format_stamp: "Formatted at 2026-09-26 16:46:31 on dist-test-slave-n1hw"
server_key: "1dab491dcc51c32f2d334ee9eec089dd"
server_key_iv: "3eedf5b8f71ffb2d8dd5171362764f17"
server_key_version: "encryptionkey@0"
I20260926 16:46:31.230803 10315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:31.235193 10315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:31.237921 10315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:31.238055 10315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:31.244932 10315 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35941
I20260926 16:46:31.244958 10383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35941 every 8 connection(s)
I20260926 16:46:31.245817 10315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:46:31.248445 10384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:31.251406 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 10315
I20260926 16:46:31.251544 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:46:31.251853 21553 external_mini_cluster.cc:1428] Setting key 37816337e67be905071964c3c4eaa3f7
I20260926 16:46:31.254585 10384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23e6ad8e198c4aad99ac6d3660f49612: Bootstrap starting.
I20260926 16:46:31.256948 10384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23e6ad8e198c4aad99ac6d3660f49612: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:31.257157 10387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:31.253259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:44379 (local address 127.0.0.1:35941)
0926 16:46:31.253694 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:31.253711 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:31.253745 (+ 34us) server_negotiation.cc:407] Connection header received
0926 16:46:31.254410 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:31.254427 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:31.254720 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:31.255062 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:31.255883 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:31.255888 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:31.255900 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:46:31.255915 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:31.255970 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:46:31.255991 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:31.256171 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:31.256242 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 16:46:31.256406 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":140,"threads_started":1}
I20260926 16:46:31.257881 10384 log.cc:824] T 00000000000000000000000000000000 P 23e6ad8e198c4aad99ac6d3660f49612: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:31.260064 10384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23e6ad8e198c4aad99ac6d3660f49612: No bootstrap required, opened a new log
I20260926 16:46:31.260214 21553 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:46:31.263985 10387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:31.262242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:54651 (local address 127.0.0.1:35941)
0926 16:46:31.262369 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:31.262373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:31.262580 (+ 207us) server_negotiation.cc:407] Connection header received
0926 16:46:31.262694 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:31.262697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:31.262730 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:31.262796 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:31.263555 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:31.263558 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:31.263560 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:46:31.263570 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:31.263606 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:46:31.263616 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:31.263696 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:31.263810 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 16:46:31.263842 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 16:46:31.264351 10384 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23e6ad8e198c4aad99ac6d3660f49612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23e6ad8e198c4aad99ac6d3660f49612" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35941 } }
I20260926 16:46:31.264590 10384 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23e6ad8e198c4aad99ac6d3660f49612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:46:31.264647 10384 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23e6ad8e198c4aad99ac6d3660f49612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23e6ad8e198c4aad99ac6d3660f49612, State: Initialized, Role: FOLLOWER
I20260926 16:46:31.265064 10384 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23e6ad8e198c4aad99ac6d3660f49612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23e6ad8e198c4aad99ac6d3660f49612" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35941 } }
I20260926 16:46:31.265216 10384 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23e6ad8e198c4aad99ac6d3660f49612 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:46:31.265293 10384 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23e6ad8e198c4aad99ac6d3660f49612 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:46:31.265390 10384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23e6ad8e198c4aad99ac6d3660f49612 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:31.266352 10384 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23e6ad8e198c4aad99ac6d3660f49612 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23e6ad8e198c4aad99ac6d3660f49612" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35941 } }
I20260926 16:46:31.266685 10384 leader_election.cc:302] T 00000000000000000000000000000000 P 23e6ad8e198c4aad99ac6d3660f49612 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23e6ad8e198c4aad99ac6d3660f49612; no voters:
I20260926 16:46:31.267010 10384 leader_election.cc:288] T 00000000000000000000000000000000 P 23e6ad8e198c4aad99ac6d3660f49612 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:46:31.267138 10394 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23e6ad8e198c4aad99ac6d3660f49612 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:46:31.267376 10394 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23e6ad8e198c4aad99ac6d3660f49612 [term 1 LEADER]: Becoming Leader. State: Replica: 23e6ad8e198c4aad99ac6d3660f49612, State: Running, Role: LEADER
I20260926 16:46:31.267750 10394 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23e6ad8e198c4aad99ac6d3660f49612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23e6ad8e198c4aad99ac6d3660f49612" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35941 } }
I20260926 16:46:31.268251 10384 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23e6ad8e198c4aad99ac6d3660f49612 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:46:31.269359 10395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23e6ad8e198c4aad99ac6d3660f49612 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23e6ad8e198c4aad99ac6d3660f49612" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23e6ad8e198c4aad99ac6d3660f49612" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35941 } } }
I20260926 16:46:31.269487 10395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23e6ad8e198c4aad99ac6d3660f49612 [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:31.269883 10403 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:46:31.269865 10396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23e6ad8e198c4aad99ac6d3660f49612 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23e6ad8e198c4aad99ac6d3660f49612. Latest consensus state: current_term: 1 leader_uuid: "23e6ad8e198c4aad99ac6d3660f49612" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23e6ad8e198c4aad99ac6d3660f49612" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35941 } } }
I20260926 16:46:31.270077 10396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23e6ad8e198c4aad99ac6d3660f49612 [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:31.272478 10403 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:46:31.277989 10403 catalog_manager.cc:1347] Generated new cluster ID: dcec0d9027244b4a9e59f690acb216ca
I20260926 16:46:31.278057 10403 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:46:31.297514 10403 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:46:31.297886 10403 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:46:31.302572 10403 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23e6ad8e198c4aad99ac6d3660f49612: Generated new TSK 0
I20260926 16:46:31.303362 10403 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:46:31.308493 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 10315
2026-09-26T16:46:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1391 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440924937351-21553-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:46:31 dist-test-slave-n1hw krb5kdc[10415](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:46:31 dist-test-slave-n1hw krb5kdc[10415](info): set up 2 sockets
Sep 26 16:46:31 dist-test-slave-n1hw krb5kdc[10415](info): commencing operation
krb5kdc: starting...
W20260926 16:46:32.364202 21553 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Sep 26 16:46:32 dist-test-slave-n1hw krb5kdc[10415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:46:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:46:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440924937351-21553-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440924937351-21553-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 16:46:32.490309 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40023
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440924937351-21553-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:46:32.600198 10428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:32.600509 10428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:32.600600 10428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:32.604585 10428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:46:32.604682 10428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:32.604722 10428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:32.604787 10428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:46:32.604844 10428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:46:32.610002 10428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40023
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440924937351-21553-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:46:32.611045 10428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:32.612071 10428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:32.620503 10434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:32.620575 10436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:32.620703 10428 server_base.cc:1034] running on GCE node
W20260926 16:46:32.620535 10433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:32.621371 10428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:32.622512 10428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:32.623667 10428 hybrid_clock.cc:648] HybridClock initialized: now 1790441192623636 us; error 45 us; skew 500 ppm
Sep 26 16:46:32 dist-test-slave-n1hw krb5kdc[10415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:46:32.626508 10428 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 16:46:32.627337 10428 webserver.cc:460] Webserver started at http://127.0.0.1:44583/ using document root <none> and password file <none>
I20260926 16:46:32.627857 10428 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:32.627912 10428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:32.628083 10428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:46:32.630120 10428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "81db07cf94504f51bb1c8c9332ab00a3"
format_stamp: "Formatted at 2026-09-26 16:46:32 on dist-test-slave-n1hw"
server_key: "b14eb8c306fc91213687d185e75a09fb"
server_key_iv: "03a3792bb24e44e88faf04bb9a8bc3b3"
server_key_version: "encryptionkey@0"
I20260926 16:46:32.630631 10428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "81db07cf94504f51bb1c8c9332ab00a3"
format_stamp: "Formatted at 2026-09-26 16:46:32 on dist-test-slave-n1hw"
server_key: "b14eb8c306fc91213687d185e75a09fb"
server_key_iv: "03a3792bb24e44e88faf04bb9a8bc3b3"
server_key_version: "encryptionkey@0"
I20260926 16:46:32.634372 10428 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:46:32.636945 10443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:32.638283 10428 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:46:32.638392 10428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "81db07cf94504f51bb1c8c9332ab00a3"
format_stamp: "Formatted at 2026-09-26 16:46:32 on dist-test-slave-n1hw"
server_key: "b14eb8c306fc91213687d185e75a09fb"
server_key_iv: "03a3792bb24e44e88faf04bb9a8bc3b3"
server_key_version: "encryptionkey@0"
I20260926 16:46:32.638484 10428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:32.648126 10428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:32.650954 10428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:32.651096 10428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:32.657924 10428 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40023
I20260926 16:46:32.658008 10495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40023 every 8 connection(s)
I20260926 16:46:32.658879 10428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:46:32.661563 10496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:32.666620 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 10428
I20260926 16:46:32.666728 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:46:32.667016 21553 external_mini_cluster.cc:1428] Setting key 9b6492e92cd6bb0b1cadfbafcd7023d1
I20260926 16:46:32.667074 10496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81db07cf94504f51bb1c8c9332ab00a3: Bootstrap starting.
I20260926 16:46:32.669551 10496 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81db07cf94504f51bb1c8c9332ab00a3: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:32.670385 10496 log.cc:824] T 00000000000000000000000000000000 P 81db07cf94504f51bb1c8c9332ab00a3: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:32.672374 10496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81db07cf94504f51bb1c8c9332ab00a3: No bootstrap required, opened a new log
Sep 26 16:46:32 dist-test-slave-n1hw krb5kdc[10415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 16:46:32.676775 10496 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81db07cf94504f51bb1c8c9332ab00a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81db07cf94504f51bb1c8c9332ab00a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40023 } }
I20260926 16:46:32.677023 10496 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81db07cf94504f51bb1c8c9332ab00a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:46:32.677109 10496 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81db07cf94504f51bb1c8c9332ab00a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81db07cf94504f51bb1c8c9332ab00a3, State: Initialized, Role: FOLLOWER
I20260926 16:46:32.677598 10496 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81db07cf94504f51bb1c8c9332ab00a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81db07cf94504f51bb1c8c9332ab00a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40023 } }
I20260926 16:46:32.677783 10496 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81db07cf94504f51bb1c8c9332ab00a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:46:32.677878 10496 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81db07cf94504f51bb1c8c9332ab00a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:46:32.677983 10496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81db07cf94504f51bb1c8c9332ab00a3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:32.678900 10496 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81db07cf94504f51bb1c8c9332ab00a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81db07cf94504f51bb1c8c9332ab00a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40023 } }
I20260926 16:46:32.679250 10496 leader_election.cc:302] T 00000000000000000000000000000000 P 81db07cf94504f51bb1c8c9332ab00a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81db07cf94504f51bb1c8c9332ab00a3; no voters:
I20260926 16:46:32.679617 10496 leader_election.cc:288] T 00000000000000000000000000000000 P 81db07cf94504f51bb1c8c9332ab00a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:46:32.679708 10501 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81db07cf94504f51bb1c8c9332ab00a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:46:32.679944 10501 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81db07cf94504f51bb1c8c9332ab00a3 [term 1 LEADER]: Becoming Leader. State: Replica: 81db07cf94504f51bb1c8c9332ab00a3, State: Running, Role: LEADER
I20260926 16:46:32.680301 10501 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81db07cf94504f51bb1c8c9332ab00a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81db07cf94504f51bb1c8c9332ab00a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40023 } }
I20260926 16:46:32.681015 10496 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81db07cf94504f51bb1c8c9332ab00a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:46:32.682032 10502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81db07cf94504f51bb1c8c9332ab00a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81db07cf94504f51bb1c8c9332ab00a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81db07cf94504f51bb1c8c9332ab00a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40023 } } }
I20260926 16:46:32.682242 10503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81db07cf94504f51bb1c8c9332ab00a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81db07cf94504f51bb1c8c9332ab00a3. Latest consensus state: current_term: 1 leader_uuid: "81db07cf94504f51bb1c8c9332ab00a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81db07cf94504f51bb1c8c9332ab00a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40023 } } }
I20260926 16:46:32.682425 10502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81db07cf94504f51bb1c8c9332ab00a3 [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:32.682525 10503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81db07cf94504f51bb1c8c9332ab00a3 [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:32.682631 10499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:32.668362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:52309 (local address 127.0.0.1:40023)
0926 16:46:32.668776 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:32.668791 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:32.668830 (+ 39us) server_negotiation.cc:407] Connection header received
0926 16:46:32.669447 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:32.669463 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:32.669912 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:32.670220 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:32.671112 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:32.672113 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:32.673248 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:32.673708 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:32.673943 (+ 235us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:32.676867 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:32.676872 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:32.676886 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:32.676913 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:32.678914 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:32.679451 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:32.679457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:32.679465 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:32.679582 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:32.680160 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:32.680163 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:32.680165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:32.680686 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:32.680908 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:32.681251 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 16:46:32.681489 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":139,"threads_started":1}
I20260926 16:46:32.682992 10512 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:46:32.685734 21553 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:46:32.686856 10512 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:46:32.692987 10512 catalog_manager.cc:1347] Generated new cluster ID: f6d862d9c5544525817bca43ad2b6472
I20260926 16:46:32.693064 10512 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:46:32.696470 10499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:32.687694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:40631 (local address 127.0.0.1:40023)
0926 16:46:32.687816 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:32.687821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:32.688058 (+ 237us) server_negotiation.cc:407] Connection header received
0926 16:46:32.688180 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:32.688183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:32.688234 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:32.688320 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:32.689036 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:32.689881 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:32.690918 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:32.691180 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:32.691300 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:32.692951 (+ 1651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:32.692955 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:32.692958 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:32.692982 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:32.694599 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:32.695072 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:32.695076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:32.695078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:32.695161 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:32.695490 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:32.695494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:32.695495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:32.695666 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:32.695837 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:32.696103 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 16:46:32.696194 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 16:46:32.707536 10512 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:46:32.708907 10512 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:46:32.719108 10512 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81db07cf94504f51bb1c8c9332ab00a3: Generated new TSK 0
I20260926 16:46:32.719833 10512 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:46:32.726974 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 10428
W20260926 16:46:32.732437 10413 connection.cc:535] client connection to 127.0.0.1:40023 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40023): Connection reset by peer (error 104)
2026-09-26T16:46:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1418 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440924937351-21553-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:46:32 dist-test-slave-n1hw krb5kdc[10528](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:46:32 dist-test-slave-n1hw krb5kdc[10528](info): set up 2 sockets
Sep 26 16:46:32 dist-test-slave-n1hw krb5kdc[10528](info): commencing operation
krb5kdc: starting...
W20260926 16:46:33.772099 21553 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Sep 26 16:46:33 dist-test-slave-n1hw krb5kdc[10528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:46:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:46:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440924937351-21553-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440924937351-21553-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 16:46:33.898306 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36365
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440924937351-21553-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:46:34.008966 10541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:34.009231 10541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:34.009272 10541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:34.013229 10541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:46:34.013300 10541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:34.013320 10541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:34.013336 10541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:46:34.013351 10541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:46:34.018671 10541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36365
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440924937351-21553-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:46:34.019677 10541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:34.020628 10541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:34.029042 10547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:34.029052 10546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:34.029042 10549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:34.029487 10541 server_base.cc:1034] running on GCE node
I20260926 16:46:34.030125 10541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:34.031226 10541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:34.032429 10541 hybrid_clock.cc:648] HybridClock initialized: now 1790441194032401 us; error 38 us; skew 500 ppm
Sep 26 16:46:34 dist-test-slave-n1hw krb5kdc[10528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441194, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:46:34.035476 10541 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 16:46:34.036387 10541 webserver.cc:460] Webserver started at http://127.0.0.1:43505/ using document root <none> and password file <none>
I20260926 16:46:34.036960 10541 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:34.037050 10541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:34.037353 10541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:46:34.039462 10541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "5beadff6efa042bfa91f7589afced2bb"
format_stamp: "Formatted at 2026-09-26 16:46:34 on dist-test-slave-n1hw"
server_key: "2359628831dd3c4b2c24710e3f6a1745"
server_key_iv: "b2f4785cb53ef4f0ab1c695a6201be11"
server_key_version: "encryptionkey@0"
I20260926 16:46:34.040046 10541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "5beadff6efa042bfa91f7589afced2bb"
format_stamp: "Formatted at 2026-09-26 16:46:34 on dist-test-slave-n1hw"
server_key: "2359628831dd3c4b2c24710e3f6a1745"
server_key_iv: "b2f4785cb53ef4f0ab1c695a6201be11"
server_key_version: "encryptionkey@0"
I20260926 16:46:34.043838 10541 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260926 16:46:34.046489 10556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:34.047879 10541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:46:34.048035 10541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "5beadff6efa042bfa91f7589afced2bb"
format_stamp: "Formatted at 2026-09-26 16:46:34 on dist-test-slave-n1hw"
server_key: "2359628831dd3c4b2c24710e3f6a1745"
server_key_iv: "b2f4785cb53ef4f0ab1c695a6201be11"
server_key_version: "encryptionkey@0"
I20260926 16:46:34.048156 10541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:34.067067 10541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:34.070806 10541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:34.071033 10541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:34.078531 10541 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36365
I20260926 16:46:34.078622 10608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36365 every 8 connection(s)
I20260926 16:46:34.079519 10541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:46:34.082396 10609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:34.085106 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 10541
I20260926 16:46:34.085273 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:46:34.085592 21553 external_mini_cluster.cc:1428] Setting key 097348a21bf71661060e5b2415403d6f
I20260926 16:46:34.088196 10609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5beadff6efa042bfa91f7589afced2bb: Bootstrap starting.
I20260926 16:46:34.090670 10609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5beadff6efa042bfa91f7589afced2bb: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:34.091513 10609 log.cc:824] T 00000000000000000000000000000000 P 5beadff6efa042bfa91f7589afced2bb: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:34.093277 10609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5beadff6efa042bfa91f7589afced2bb: No bootstrap required, opened a new log
I20260926 16:46:34.094556 10612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:34.086989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:47817 (local address 127.0.0.1:36365)
0926 16:46:34.087423 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:34.087438 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:34.087472 (+ 34us) server_negotiation.cc:407] Connection header received
0926 16:46:34.088071 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:34.088087 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:34.088481 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:34.088757 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:34.089071 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:34.090118 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:34.091293 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:34.091826 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:34.091988 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:34.092895 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:34.092900 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:34.092913 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:46:34.093277 (+ 364us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:34.093337 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:46:34.093360 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:34.093537 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:34.093686 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 16:46:34.093852 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":146,"threads_started":1}
I20260926 16:46:34.096937 10609 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5beadff6efa042bfa91f7589afced2bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5beadff6efa042bfa91f7589afced2bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36365 } }
I20260926 16:46:34.097163 10609 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5beadff6efa042bfa91f7589afced2bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:46:34.097254 10609 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5beadff6efa042bfa91f7589afced2bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beadff6efa042bfa91f7589afced2bb, State: Initialized, Role: FOLLOWER
I20260926 16:46:34.097353 21553 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:46:34.097734 10609 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5beadff6efa042bfa91f7589afced2bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5beadff6efa042bfa91f7589afced2bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36365 } }
I20260926 16:46:34.097931 10609 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5beadff6efa042bfa91f7589afced2bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:46:34.097999 10609 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5beadff6efa042bfa91f7589afced2bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:46:34.098091 10609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5beadff6efa042bfa91f7589afced2bb [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:34.099089 10609 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5beadff6efa042bfa91f7589afced2bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5beadff6efa042bfa91f7589afced2bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36365 } }
I20260926 16:46:34.099457 10609 leader_election.cc:302] T 00000000000000000000000000000000 P 5beadff6efa042bfa91f7589afced2bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5beadff6efa042bfa91f7589afced2bb; no voters:
I20260926 16:46:34.099772 10609 leader_election.cc:288] T 00000000000000000000000000000000 P 5beadff6efa042bfa91f7589afced2bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:46:34.099913 10619 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5beadff6efa042bfa91f7589afced2bb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:46:34.100164 10619 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5beadff6efa042bfa91f7589afced2bb [term 1 LEADER]: Becoming Leader. State: Replica: 5beadff6efa042bfa91f7589afced2bb, State: Running, Role: LEADER
I20260926 16:46:34.100531 10619 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5beadff6efa042bfa91f7589afced2bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5beadff6efa042bfa91f7589afced2bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36365 } }
I20260926 16:46:34.101018 10609 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5beadff6efa042bfa91f7589afced2bb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:46:34.102130 10621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5beadff6efa042bfa91f7589afced2bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5beadff6efa042bfa91f7589afced2bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5beadff6efa042bfa91f7589afced2bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36365 } } }
I20260926 16:46:34.102260 10621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5beadff6efa042bfa91f7589afced2bb [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:34.102488 10621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5beadff6efa042bfa91f7589afced2bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5beadff6efa042bfa91f7589afced2bb. Latest consensus state: current_term: 1 leader_uuid: "5beadff6efa042bfa91f7589afced2bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5beadff6efa042bfa91f7589afced2bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36365 } } }
I20260926 16:46:34.102558 10621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5beadff6efa042bfa91f7589afced2bb [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:34.103900 10628 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:46:34.104380 10612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:34.099452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:45257 (local address 127.0.0.1:36365)
0926 16:46:34.099581 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:34.099585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:34.099819 (+ 234us) server_negotiation.cc:407] Connection header received
0926 16:46:34.099935 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:34.099937 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:34.099985 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:34.100076 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:34.100430 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:34.101185 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:34.102389 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:34.102764 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:34.102838 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:34.103842 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:34.103848 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:34.103852 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:46:34.103873 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:34.103904 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:46:34.103919 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:34.103987 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:34.104226 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 16:46:34.104260 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 16:46:34.107223 10628 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:46:34.112601 10628 catalog_manager.cc:1347] Generated new cluster ID: bcc7b0ad054e4ed08819ce10e7cba1d4
I20260926 16:46:34.112701 10628 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:46:34.146580 10628 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:46:34.147636 10628 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:46:34.156564 10628 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5beadff6efa042bfa91f7589afced2bb: Generated new TSK 0
I20260926 16:46:34.157286 10628 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:46:34.162972 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 10541
W20260926 16:46:34.168540 10526 connection.cc:535] client connection to 127.0.0.1:36365 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36365): Connection reset by peer (error 104)
2026-09-26T16:46:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1435 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440924937351-21553-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:46:34 dist-test-slave-n1hw krb5kdc[10640](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:46:34 dist-test-slave-n1hw krb5kdc[10640](info): set up 2 sockets
Sep 26 16:46:34 dist-test-slave-n1hw krb5kdc[10640](info): commencing operation
krb5kdc: starting...
W20260926 16:46:35.212006 21553 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Sep 26 16:46:35 dist-test-slave-n1hw krb5kdc[10640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:46:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:46:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440924937351-21553-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440924937351-21553-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 16:46:35.337001 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40895
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440924937351-21553-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:46:35.449430 10653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:35.449726 10653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:35.449787 10653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:35.454103 10653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:46:35.454161 10653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:35.454181 10653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:35.454197 10653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:46:35.454212 10653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:46:35.460041 10653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40895
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440924937351-21553-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:46:35.461171 10653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:35.462299 10653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:35.471128 10659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:35.471158 10658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:35.471143 10661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:35.471201 10653 server_base.cc:1034] running on GCE node
I20260926 16:46:35.472051 10653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:35.473187 10653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:35.474375 10653 hybrid_clock.cc:648] HybridClock initialized: now 1790441195474345 us; error 45 us; skew 500 ppm
Sep 26 16:46:35 dist-test-slave-n1hw krb5kdc[10640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:46:35.483592 10653 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 16:46:35.484561 10653 webserver.cc:460] Webserver started at http://127.0.0.1:40709/ using document root <none> and password file <none>
I20260926 16:46:35.485178 10653 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:35.485272 10653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:35.485518 10653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:46:35.487645 10653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "e72ce8c574ad4c3a9472c753f8740a03"
format_stamp: "Formatted at 2026-09-26 16:46:35 on dist-test-slave-n1hw"
server_key: "e0c61399acd7bc3d46acedf757f60f92"
server_key_iv: "e3ccd29242e3154bebcbcbe07cac49dd"
server_key_version: "encryptionkey@0"
I20260926 16:46:35.488265 10653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "e72ce8c574ad4c3a9472c753f8740a03"
format_stamp: "Formatted at 2026-09-26 16:46:35 on dist-test-slave-n1hw"
server_key: "e0c61399acd7bc3d46acedf757f60f92"
server_key_iv: "e3ccd29242e3154bebcbcbe07cac49dd"
server_key_version: "encryptionkey@0"
I20260926 16:46:35.492112 10653 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 16:46:35.494726 10668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:35.496208 10653 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:46:35.496342 10653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "e72ce8c574ad4c3a9472c753f8740a03"
format_stamp: "Formatted at 2026-09-26 16:46:35 on dist-test-slave-n1hw"
server_key: "e0c61399acd7bc3d46acedf757f60f92"
server_key_iv: "e3ccd29242e3154bebcbcbe07cac49dd"
server_key_version: "encryptionkey@0"
I20260926 16:46:35.496475 10653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:35.501236 10653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:35.504160 10653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:35.504350 10653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:35.511371 10653 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40895
I20260926 16:46:35.511413 10721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40895 every 8 connection(s)
I20260926 16:46:35.512279 10653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:46:35.513477 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 10653
I20260926 16:46:35.513610 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:46:35.514021 21553 external_mini_cluster.cc:1428] Setting key caec39b386fd96176c86c7dd7ddc25b8
I20260926 16:46:35.515352 10722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:35.519728 10725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:35.515467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:58513 (local address 127.0.0.1:40895)
0926 16:46:35.515907 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:35.515925 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:35.515966 (+ 41us) server_negotiation.cc:407] Connection header received
0926 16:46:35.516784 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:35.516808 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:35.517178 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:35.517542 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:35.518317 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:35.518323 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:35.518339 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:46:35.518350 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:35.518416 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:46:35.518442 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:35.518591 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:35.518659 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 16:46:35.518836 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":108,"threads_started":1}
I20260926 16:46:35.522038 10722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e72ce8c574ad4c3a9472c753f8740a03: Bootstrap starting.
I20260926 16:46:35.522501 21553 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:46:35.524741 10722 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e72ce8c574ad4c3a9472c753f8740a03: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:35.525758 10722 log.cc:824] T 00000000000000000000000000000000 P e72ce8c574ad4c3a9472c753f8740a03: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:35.526490 10725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:35.524741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:46077 (local address 127.0.0.1:40895)
0926 16:46:35.524911 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:35.524915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:35.525080 (+ 165us) server_negotiation.cc:407] Connection header received
0926 16:46:35.525199 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:35.525201 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:35.525228 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:35.525291 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:35.526095 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:35.526098 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:35.526101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:46:35.526112 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:35.526137 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:46:35.526155 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:35.526242 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:35.526332 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 16:46:35.526373 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 16:46:35.527876 10722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e72ce8c574ad4c3a9472c753f8740a03: No bootstrap required, opened a new log
I20260926 16:46:35.531230 10722 raft_consensus.cc:348] T 00000000000000000000000000000000 P e72ce8c574ad4c3a9472c753f8740a03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e72ce8c574ad4c3a9472c753f8740a03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40895 } }
I20260926 16:46:35.531466 10722 raft_consensus.cc:374] T 00000000000000000000000000000000 P e72ce8c574ad4c3a9472c753f8740a03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:46:35.531502 10722 raft_consensus.cc:729] T 00000000000000000000000000000000 P e72ce8c574ad4c3a9472c753f8740a03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ce8c574ad4c3a9472c753f8740a03, State: Initialized, Role: FOLLOWER
I20260926 16:46:35.531908 10722 consensus_queue.cc:260] T 00000000000000000000000000000000 P e72ce8c574ad4c3a9472c753f8740a03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e72ce8c574ad4c3a9472c753f8740a03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40895 } }
I20260926 16:46:35.532024 10722 raft_consensus.cc:388] T 00000000000000000000000000000000 P e72ce8c574ad4c3a9472c753f8740a03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:46:35.532070 10722 raft_consensus.cc:482] T 00000000000000000000000000000000 P e72ce8c574ad4c3a9472c753f8740a03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:46:35.532137 10722 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e72ce8c574ad4c3a9472c753f8740a03 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:35.533133 10722 raft_consensus.cc:504] T 00000000000000000000000000000000 P e72ce8c574ad4c3a9472c753f8740a03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e72ce8c574ad4c3a9472c753f8740a03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40895 } }
I20260926 16:46:35.533505 10722 leader_election.cc:302] T 00000000000000000000000000000000 P e72ce8c574ad4c3a9472c753f8740a03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e72ce8c574ad4c3a9472c753f8740a03; no voters:
I20260926 16:46:35.533859 10722 leader_election.cc:288] T 00000000000000000000000000000000 P e72ce8c574ad4c3a9472c753f8740a03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:46:35.533986 10732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e72ce8c574ad4c3a9472c753f8740a03 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:46:35.534242 10732 raft_consensus.cc:686] T 00000000000000000000000000000000 P e72ce8c574ad4c3a9472c753f8740a03 [term 1 LEADER]: Becoming Leader. State: Replica: e72ce8c574ad4c3a9472c753f8740a03, State: Running, Role: LEADER
I20260926 16:46:35.534636 10732 consensus_queue.cc:237] T 00000000000000000000000000000000 P e72ce8c574ad4c3a9472c753f8740a03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e72ce8c574ad4c3a9472c753f8740a03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40895 } }
I20260926 16:46:35.535156 10722 sys_catalog.cc:564] T 00000000000000000000000000000000 P e72ce8c574ad4c3a9472c753f8740a03 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:46:35.536428 10733 sys_catalog.cc:455] T 00000000000000000000000000000000 P e72ce8c574ad4c3a9472c753f8740a03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e72ce8c574ad4c3a9472c753f8740a03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e72ce8c574ad4c3a9472c753f8740a03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40895 } } }
I20260926 16:46:35.536628 10733 sys_catalog.cc:458] T 00000000000000000000000000000000 P e72ce8c574ad4c3a9472c753f8740a03 [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:35.536739 10734 sys_catalog.cc:455] T 00000000000000000000000000000000 P e72ce8c574ad4c3a9472c753f8740a03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e72ce8c574ad4c3a9472c753f8740a03. Latest consensus state: current_term: 1 leader_uuid: "e72ce8c574ad4c3a9472c753f8740a03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e72ce8c574ad4c3a9472c753f8740a03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40895 } } }
I20260926 16:46:35.536839 10734 sys_catalog.cc:458] T 00000000000000000000000000000000 P e72ce8c574ad4c3a9472c753f8740a03 [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:35.537182 10743 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:46:35.539883 10743 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:46:35.544986 10743 catalog_manager.cc:1347] Generated new cluster ID: e6adab837c5f4566b8a071e0f1dc8499
I20260926 16:46:35.545051 10743 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:46:35.567178 10743 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:46:35.567579 10743 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:46:35.572733 10743 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e72ce8c574ad4c3a9472c753f8740a03: Generated new TSK 0
I20260926 16:46:35.573549 10743 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:46:35.578858 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 10653
2026-09-26T16:46:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1415 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440924937351-21553-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:46:35 dist-test-slave-n1hw krb5kdc[10753](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:46:35 dist-test-slave-n1hw krb5kdc[10753](info): set up 2 sockets
Sep 26 16:46:35 dist-test-slave-n1hw krb5kdc[10753](info): commencing operation
krb5kdc: starting...
W20260926 16:46:36.628136 21553 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Sep 26 16:46:36 dist-test-slave-n1hw krb5kdc[10753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:46:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:46:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
I20260926 16:46:36.751106 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:38973
--webserver_interface=127.21.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:43873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:38973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:46:36.863416 10766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:36.863711 10766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:36.863756 10766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:36.868001 10766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:46:36.868067 10766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:36.868088 10766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:36.868104 10766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:46:36.868119 10766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:46:36.873525 10766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:43873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:38973
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:38973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:46:36.874634 10766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:36.875692 10766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:36.884243 10771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:36.884274 10766 server_base.cc:1034] running on GCE node
W20260926 16:46:36.884224 10772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:36.884291 10774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:36.885087 10766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:36.886267 10766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:36.887454 10766 hybrid_clock.cc:648] HybridClock initialized: now 1790441196887422 us; error 45 us; skew 500 ppm
Sep 26 16:46:36 dist-test-slave-n1hw krb5kdc[10753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441196, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:46:36.890370 10766 init.cc:377] Logged in from keytab as kudu/127.21.12.126@KRBTEST.COM (short username kudu)
I20260926 16:46:36.891280 10766 webserver.cc:460] Webserver started at http://127.21.12.126:33313/ using document root <none> and password file <none>
I20260926 16:46:36.891881 10766 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:36.891973 10766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:36.892204 10766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:46:36.894309 10766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "f7c84c444d2e4927afbeee846a854679"
format_stamp: "Formatted at 2026-09-26 16:46:36 on dist-test-slave-n1hw"
server_key: "757166940590c0f9065cc7edabede3da"
server_key_iv: "8314a4f484784b64519816889014c672"
server_key_version: "encryptionkey@0"
I20260926 16:46:36.894896 10766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "f7c84c444d2e4927afbeee846a854679"
format_stamp: "Formatted at 2026-09-26 16:46:36 on dist-test-slave-n1hw"
server_key: "757166940590c0f9065cc7edabede3da"
server_key_iv: "8314a4f484784b64519816889014c672"
server_key_version: "encryptionkey@0"
I20260926 16:46:36.898610 10766 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:46:36.901006 10781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:36.902364 10766 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:46:36.902508 10766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "f7c84c444d2e4927afbeee846a854679"
format_stamp: "Formatted at 2026-09-26 16:46:36 on dist-test-slave-n1hw"
server_key: "757166940590c0f9065cc7edabede3da"
server_key_iv: "8314a4f484784b64519816889014c672"
server_key_version: "encryptionkey@0"
I20260926 16:46:36.902623 10766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:36.916731 10766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:36.920183 10766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:36.920396 10766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:36.927436 10766 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.126:38973
I20260926 16:46:36.927462 10833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.126:38973 every 8 connection(s)
I20260926 16:46:36.928390 10766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:46:36.931450 10834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:36.937379 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 10766
I20260926 16:46:36.937530 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:46:36.937692 10834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7c84c444d2e4927afbeee846a854679: Bootstrap starting.
I20260926 16:46:36.937903 21553 external_mini_cluster.cc:1428] Setting key 5f5b4cbe2fbaead32c76edc781c7c9f0
I20260926 16:46:36.940528 10834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7c84c444d2e4927afbeee846a854679: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:36.941305 10834 log.cc:824] T 00000000000000000000000000000000 P f7c84c444d2e4927afbeee846a854679: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:36.943308 10834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7c84c444d2e4927afbeee846a854679: No bootstrap required, opened a new log
Sep 26 16:46:36 dist-test-slave-n1hw krb5kdc[10753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:46:36.947237 10834 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7c84c444d2e4927afbeee846a854679 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c84c444d2e4927afbeee846a854679" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 38973 } }
I20260926 16:46:36.947499 10834 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7c84c444d2e4927afbeee846a854679 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:46:36.947589 10834 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7c84c444d2e4927afbeee846a854679 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c84c444d2e4927afbeee846a854679, State: Initialized, Role: FOLLOWER
I20260926 16:46:36.948154 10834 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7c84c444d2e4927afbeee846a854679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c84c444d2e4927afbeee846a854679" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 38973 } }
I20260926 16:46:36.948351 10834 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7c84c444d2e4927afbeee846a854679 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:46:36.948432 10834 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7c84c444d2e4927afbeee846a854679 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:46:36.948541 10834 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7c84c444d2e4927afbeee846a854679 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:36.949455 10834 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7c84c444d2e4927afbeee846a854679 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c84c444d2e4927afbeee846a854679" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 38973 } }
I20260926 16:46:36.949885 10834 leader_election.cc:302] T 00000000000000000000000000000000 P f7c84c444d2e4927afbeee846a854679 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7c84c444d2e4927afbeee846a854679; no voters:
I20260926 16:46:36.950237 10834 leader_election.cc:288] T 00000000000000000000000000000000 P f7c84c444d2e4927afbeee846a854679 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:46:36.950317 10839 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7c84c444d2e4927afbeee846a854679 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:46:36.950531 10839 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7c84c444d2e4927afbeee846a854679 [term 1 LEADER]: Becoming Leader. State: Replica: f7c84c444d2e4927afbeee846a854679, State: Running, Role: LEADER
I20260926 16:46:36.950904 10839 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7c84c444d2e4927afbeee846a854679 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c84c444d2e4927afbeee846a854679" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 38973 } }
I20260926 16:46:36.951665 10834 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7c84c444d2e4927afbeee846a854679 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:46:36.952939 10841 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7c84c444d2e4927afbeee846a854679 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7c84c444d2e4927afbeee846a854679" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c84c444d2e4927afbeee846a854679" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 38973 } } }
I20260926 16:46:36.953071 10841 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7c84c444d2e4927afbeee846a854679 [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:36.953099 10837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:36.939397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46431 (local address 127.21.12.126:38973)
0926 16:46:36.939858 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:36.939873 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:36.939909 (+ 36us) server_negotiation.cc:407] Connection header received
0926 16:46:36.940545 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:36.940562 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:36.941001 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:36.941345 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:36.942205 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:36.943165 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:36.944093 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:36.944685 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:36.944858 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:36.947462 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:36.947473 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:36.947486 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:36.947534 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:36.949478 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:36.950059 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:36.950068 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:36.950077 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:36.950203 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:36.950570 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:36.950575 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:36.950577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:36.951115 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:36.951351 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:36.951687 (+ 336us) server_negotiation.cc:299] Negotiation successful
0926 16:46:36.952068 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":129,"threads_started":1}
I20260926 16:46:36.952940 10840 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7c84c444d2e4927afbeee846a854679 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7c84c444d2e4927afbeee846a854679. Latest consensus state: current_term: 1 leader_uuid: "f7c84c444d2e4927afbeee846a854679" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c84c444d2e4927afbeee846a854679" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 38973 } } }
I20260926 16:46:36.953368 10840 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7c84c444d2e4927afbeee846a854679 [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:36.956521 21553 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260926 16:46:36.957144 10854 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f7c84c444d2e4927afbeee846a854679: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 16:46:36.957244 10854 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 16:46:36.957338 10848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:46:36.958190 10848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:46:36.964210 10848 catalog_manager.cc:1347] Generated new cluster ID: 8a306ec45ec54ee1a3927a18b0ff1e14
I20260926 16:46:36.964275 10848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:46:36.967932 10837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:36.958759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55357 (local address 127.21.12.126:38973)
0926 16:46:36.958867 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:36.958872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:36.959133 (+ 261us) server_negotiation.cc:407] Connection header received
0926 16:46:36.959218 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:36.959221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:36.959269 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:36.959366 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:36.960136 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:36.960937 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:36.962051 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:36.962429 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:36.962537 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:36.964253 (+ 1716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:36.964257 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:36.964259 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:36.964285 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:36.966000 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:36.966473 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:36.966476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:36.966478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:36.966562 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:36.966896 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:36.966900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:36.966902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:36.967078 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:36.967284 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:36.967555 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 16:46:36.967645 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 16:46:36.983829 10848 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:46:36.985168 10848 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:46:36.995102 10848 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7c84c444d2e4927afbeee846a854679: Generated new TSK 0
I20260926 16:46:36.995954 10848 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:46:37.011041 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 10766
W20260926 16:46:37.017168 10751 connection.cc:535] client connection to 127.21.12.126:38973 recv error: Network error: failed to read from TLS socket (remote: 127.21.12.126:38973): Connection reset by peer (error 104)
2026-09-26T16:46:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1433 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440924937351-21553-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:46:37 dist-test-slave-n1hw krb5kdc[10865](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:46:37 dist-test-slave-n1hw krb5kdc[10865](info): set up 2 sockets
Sep 26 16:46:37 dist-test-slave-n1hw krb5kdc[10865](info): commencing operation
krb5kdc: starting...
W20260926 16:46:38.083158 21553 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Sep 26 16:46:38 dist-test-slave-n1hw krb5kdc[10865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:46:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:46:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
I20260926 16:46:38.207036 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:36527
--webserver_interface=127.21.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:40641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:36527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:46:38.319267 10878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:38.319556 10878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:38.319648 10878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:38.323720 10878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:46:38.323808 10878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:38.323843 10878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:38.323899 10878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:46:38.323932 10878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:46:38.329221 10878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:40641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:36527
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:36527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:46:38.330371 10878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:38.331387 10878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:38.340242 10886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:38.340278 10884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:38.340248 10883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:38.340663 10878 server_base.cc:1034] running on GCE node
I20260926 16:46:38.341300 10878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:38.342537 10878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:38.343739 10878 hybrid_clock.cc:648] HybridClock initialized: now 1790441198343699 us; error 51 us; skew 500 ppm
Sep 26 16:46:38 dist-test-slave-n1hw krb5kdc[10865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441198, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:46:38.346776 10878 init.cc:377] Logged in from keytab as kudu/127.21.12.126@KRBTEST.COM (short username kudu)
I20260926 16:46:38.347628 10878 webserver.cc:460] Webserver started at http://127.21.12.126:38571/ using document root <none> and password file <none>
I20260926 16:46:38.348199 10878 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:38.348261 10878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:38.348446 10878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:46:38.350513 10878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "255e85c12dc542afb3ef8517f8e53c87"
format_stamp: "Formatted at 2026-09-26 16:46:38 on dist-test-slave-n1hw"
server_key: "9aa077c31a493565e788bb681b80af0b"
server_key_iv: "accf51cc22621a58c28a44232874a806"
server_key_version: "encryptionkey@0"
I20260926 16:46:38.351087 10878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "255e85c12dc542afb3ef8517f8e53c87"
format_stamp: "Formatted at 2026-09-26 16:46:38 on dist-test-slave-n1hw"
server_key: "9aa077c31a493565e788bb681b80af0b"
server_key_iv: "accf51cc22621a58c28a44232874a806"
server_key_version: "encryptionkey@0"
I20260926 16:46:38.354764 10878 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:46:38.357158 10893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:38.358523 10878 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:46:38.358624 10878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "255e85c12dc542afb3ef8517f8e53c87"
format_stamp: "Formatted at 2026-09-26 16:46:38 on dist-test-slave-n1hw"
server_key: "9aa077c31a493565e788bb681b80af0b"
server_key_iv: "accf51cc22621a58c28a44232874a806"
server_key_version: "encryptionkey@0"
I20260926 16:46:38.358709 10878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:38.376448 10878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:38.379418 10878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:38.379573 10878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:38.386762 10878 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.126:36527
I20260926 16:46:38.386778 10945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.126:36527 every 8 connection(s)
I20260926 16:46:38.387732 10878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:46:38.390527 10946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:38.393098 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 10878
I20260926 16:46:38.393206 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:46:38.393502 21553 external_mini_cluster.cc:1428] Setting key b08a5de930631f4fcda2914231aa8521
I20260926 16:46:38.396876 10946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 255e85c12dc542afb3ef8517f8e53c87: Bootstrap starting.
I20260926 16:46:38.399209 10946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 255e85c12dc542afb3ef8517f8e53c87: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:38.399977 10946 log.cc:824] T 00000000000000000000000000000000 P 255e85c12dc542afb3ef8517f8e53c87: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:38.402531 10946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 255e85c12dc542afb3ef8517f8e53c87: No bootstrap required, opened a new log
I20260926 16:46:38.403060 10949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:38.394954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56455 (local address 127.21.12.126:36527)
0926 16:46:38.395409 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:38.395423 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:38.395457 (+ 34us) server_negotiation.cc:407] Connection header received
0926 16:46:38.396113 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:38.396131 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:38.396476 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:38.396822 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:38.397217 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:38.398244 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:38.399144 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:38.399698 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:38.399882 (+ 184us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:38.400900 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:38.400905 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:38.400918 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:46:38.401458 (+ 540us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:38.401526 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:46:38.401559 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:38.401710 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:38.401999 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 16:46:38.402128 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":152,"threads_started":1}
I20260926 16:46:38.406322 21553 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:46:38.406877 10946 raft_consensus.cc:348] T 00000000000000000000000000000000 P 255e85c12dc542afb3ef8517f8e53c87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "255e85c12dc542afb3ef8517f8e53c87" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 36527 } }
I20260926 16:46:38.407162 10946 raft_consensus.cc:374] T 00000000000000000000000000000000 P 255e85c12dc542afb3ef8517f8e53c87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:46:38.407243 10946 raft_consensus.cc:729] T 00000000000000000000000000000000 P 255e85c12dc542afb3ef8517f8e53c87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 255e85c12dc542afb3ef8517f8e53c87, State: Initialized, Role: FOLLOWER
I20260926 16:46:38.407739 10946 consensus_queue.cc:260] T 00000000000000000000000000000000 P 255e85c12dc542afb3ef8517f8e53c87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "255e85c12dc542afb3ef8517f8e53c87" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 36527 } }
I20260926 16:46:38.407927 10946 raft_consensus.cc:388] T 00000000000000000000000000000000 P 255e85c12dc542afb3ef8517f8e53c87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:46:38.408003 10946 raft_consensus.cc:482] T 00000000000000000000000000000000 P 255e85c12dc542afb3ef8517f8e53c87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:46:38.408110 10946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 255e85c12dc542afb3ef8517f8e53c87 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:38.409307 10946 raft_consensus.cc:504] T 00000000000000000000000000000000 P 255e85c12dc542afb3ef8517f8e53c87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "255e85c12dc542afb3ef8517f8e53c87" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 36527 } }
I20260926 16:46:38.409716 10946 leader_election.cc:302] T 00000000000000000000000000000000 P 255e85c12dc542afb3ef8517f8e53c87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 255e85c12dc542afb3ef8517f8e53c87; no voters:
I20260926 16:46:38.410110 10946 leader_election.cc:288] T 00000000000000000000000000000000 P 255e85c12dc542afb3ef8517f8e53c87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:46:38.410194 10957 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 255e85c12dc542afb3ef8517f8e53c87 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:46:38.410387 10957 raft_consensus.cc:686] T 00000000000000000000000000000000 P 255e85c12dc542afb3ef8517f8e53c87 [term 1 LEADER]: Becoming Leader. State: Replica: 255e85c12dc542afb3ef8517f8e53c87, State: Running, Role: LEADER
I20260926 16:46:38.410758 10957 consensus_queue.cc:237] T 00000000000000000000000000000000 P 255e85c12dc542afb3ef8517f8e53c87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "255e85c12dc542afb3ef8517f8e53c87" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 36527 } }
I20260926 16:46:38.411408 10946 sys_catalog.cc:564] T 00000000000000000000000000000000 P 255e85c12dc542afb3ef8517f8e53c87 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:46:38.411998 10957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 255e85c12dc542afb3ef8517f8e53c87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 255e85c12dc542afb3ef8517f8e53c87. Latest consensus state: current_term: 1 leader_uuid: "255e85c12dc542afb3ef8517f8e53c87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "255e85c12dc542afb3ef8517f8e53c87" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 36527 } } }
I20260926 16:46:38.412202 10957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 255e85c12dc542afb3ef8517f8e53c87 [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:38.412532 10958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 255e85c12dc542afb3ef8517f8e53c87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "255e85c12dc542afb3ef8517f8e53c87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "255e85c12dc542afb3ef8517f8e53c87" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 36527 } } }
I20260926 16:46:38.412652 10964 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:46:38.412699 10958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 255e85c12dc542afb3ef8517f8e53c87 [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:38.413950 10949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:38.408697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54681 (local address 127.21.12.126:36527)
0926 16:46:38.408823 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:38.408827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:38.409102 (+ 275us) server_negotiation.cc:407] Connection header received
0926 16:46:38.409254 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:38.409256 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:38.409299 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:38.409369 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:38.409698 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:38.410573 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:38.411607 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:38.411985 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:38.412093 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:38.413323 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:38.413329 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:38.413335 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:46:38.413350 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:38.413379 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:46:38.413396 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:38.413509 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:38.413670 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 16:46:38.413706 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 16:46:38.416517 10964 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:46:38.426152 10964 catalog_manager.cc:1347] Generated new cluster ID: b486995e5b3c4d859e7e3aa07b9692de
I20260926 16:46:38.426241 10964 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:46:38.455248 10964 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:46:38.456235 10964 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:46:38.461968 10964 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 255e85c12dc542afb3ef8517f8e53c87: Generated new TSK 0
I20260926 16:46:38.462663 10964 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:46:38.475548 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 10878
W20260926 16:46:38.481149 10863 connection.cc:535] client connection to 127.21.12.126:36527 recv error: Network error: failed to read from TLS socket (remote: 127.21.12.126:36527): Connection reset by peer (error 104)
2026-09-26T16:46:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1463 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440924937351-21553-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:46:38 dist-test-slave-n1hw krb5kdc[10978](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:46:38 dist-test-slave-n1hw krb5kdc[10978](info): set up 2 sockets
Sep 26 16:46:38 dist-test-slave-n1hw krb5kdc[10978](info): commencing operation
krb5kdc: starting...
W20260926 16:46:39.524171 21553 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Sep 26 16:46:39 dist-test-slave-n1hw krb5kdc[10978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:46:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:46:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
I20260926 16:46:39.649325 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:44739
--webserver_interface=127.21.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:38873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:44739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:46:39.759472 10991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:39.759778 10991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:39.759882 10991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:39.763999 10991 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:46:39.764081 10991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:39.764124 10991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:39.764182 10991 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:46:39.764220 10991 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:46:39.769435 10991 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:38873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:44739
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:44739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:46:39.770505 10991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:39.771545 10991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:39.779863 10999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:39.780041 10997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:39.779932 10991 server_base.cc:1034] running on GCE node
W20260926 16:46:39.779865 10996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:39.780699 10991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:39.781729 10991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:39.782996 10991 hybrid_clock.cc:648] HybridClock initialized: now 1790441199782977 us; error 32 us; skew 500 ppm
Sep 26 16:46:39 dist-test-slave-n1hw krb5kdc[10978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441199, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:46:39.785810 10991 init.cc:377] Logged in from keytab as kudu/127.21.12.126@KRBTEST.COM (short username kudu)
I20260926 16:46:39.786689 10991 webserver.cc:460] Webserver started at http://127.21.12.126:41705/ using document root <none> and password file <none>
I20260926 16:46:39.787248 10991 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:39.787338 10991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:39.787563 10991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:46:39.789549 10991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "ffa7bbe29c97454b9bf6b20fd7f9e678"
format_stamp: "Formatted at 2026-09-26 16:46:39 on dist-test-slave-n1hw"
server_key: "4cc665c532f3efff4bdbb25ed14f8147"
server_key_iv: "af19ce7860c03e8a294e9edd503e9b68"
server_key_version: "encryptionkey@0"
I20260926 16:46:39.790161 10991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "ffa7bbe29c97454b9bf6b20fd7f9e678"
format_stamp: "Formatted at 2026-09-26 16:46:39 on dist-test-slave-n1hw"
server_key: "4cc665c532f3efff4bdbb25ed14f8147"
server_key_iv: "af19ce7860c03e8a294e9edd503e9b68"
server_key_version: "encryptionkey@0"
I20260926 16:46:39.794006 10991 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:46:39.796404 11006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:39.797703 10991 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:46:39.797911 10991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "ffa7bbe29c97454b9bf6b20fd7f9e678"
format_stamp: "Formatted at 2026-09-26 16:46:39 on dist-test-slave-n1hw"
server_key: "4cc665c532f3efff4bdbb25ed14f8147"
server_key_iv: "af19ce7860c03e8a294e9edd503e9b68"
server_key_version: "encryptionkey@0"
I20260926 16:46:39.798027 10991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:39.802517 10991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:39.805354 10991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:39.805548 10991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:39.812501 10991 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.126:44739
I20260926 16:46:39.812525 11058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.126:44739 every 8 connection(s)
I20260926 16:46:39.813390 10991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:46:39.815821 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 10991
I20260926 16:46:39.815958 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:46:39.816179 11059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:39.816282 21553 external_mini_cluster.cc:1428] Setting key 66ec4fef18d9c5d561f19874fb65ab6d
I20260926 16:46:39.821803 11062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:39.817651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49589 (local address 127.21.12.126:44739)
0926 16:46:39.818151 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:39.818169 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:39.818213 (+ 44us) server_negotiation.cc:407] Connection header received
0926 16:46:39.818967 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:39.818990 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:39.819351 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:39.819737 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:39.820438 (+ 701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:39.820444 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:39.820459 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:46:39.820471 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:39.820529 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:46:39.820553 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:39.820687 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:39.820751 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 16:46:39.820897 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":127,"threads_started":1}
I20260926 16:46:39.823560 11059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffa7bbe29c97454b9bf6b20fd7f9e678: Bootstrap starting.
I20260926 16:46:39.824682 21553 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:46:39.825958 11059 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffa7bbe29c97454b9bf6b20fd7f9e678: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:39.826836 11059 log.cc:824] T 00000000000000000000000000000000 P ffa7bbe29c97454b9bf6b20fd7f9e678: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:39.828542 11062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:39.826820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34001 (local address 127.21.12.126:44739)
0926 16:46:39.826933 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:39.826937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:39.827113 (+ 176us) server_negotiation.cc:407] Connection header received
0926 16:46:39.827264 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:39.827267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:39.827295 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:39.827380 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:39.828122 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:39.828126 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:39.828129 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:46:39.828168 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:39.828191 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:46:39.828201 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:39.828309 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:39.828385 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 16:46:39.828423 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 16:46:39.828755 11059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffa7bbe29c97454b9bf6b20fd7f9e678: No bootstrap required, opened a new log
I20260926 16:46:39.832275 11059 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffa7bbe29c97454b9bf6b20fd7f9e678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa7bbe29c97454b9bf6b20fd7f9e678" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 44739 } }
I20260926 16:46:39.832496 11059 raft_consensus.cc:374] T 00000000000000000000000000000000 P ffa7bbe29c97454b9bf6b20fd7f9e678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:46:39.832532 11059 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffa7bbe29c97454b9bf6b20fd7f9e678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffa7bbe29c97454b9bf6b20fd7f9e678, State: Initialized, Role: FOLLOWER
I20260926 16:46:39.832975 11059 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffa7bbe29c97454b9bf6b20fd7f9e678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa7bbe29c97454b9bf6b20fd7f9e678" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 44739 } }
I20260926 16:46:39.833161 11059 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffa7bbe29c97454b9bf6b20fd7f9e678 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:46:39.833233 11059 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffa7bbe29c97454b9bf6b20fd7f9e678 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:46:39.833328 11059 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffa7bbe29c97454b9bf6b20fd7f9e678 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:39.834285 11059 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffa7bbe29c97454b9bf6b20fd7f9e678 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa7bbe29c97454b9bf6b20fd7f9e678" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 44739 } }
I20260926 16:46:39.834630 11059 leader_election.cc:302] T 00000000000000000000000000000000 P ffa7bbe29c97454b9bf6b20fd7f9e678 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffa7bbe29c97454b9bf6b20fd7f9e678; no voters:
I20260926 16:46:39.834957 11059 leader_election.cc:288] T 00000000000000000000000000000000 P ffa7bbe29c97454b9bf6b20fd7f9e678 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:46:39.835078 11069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffa7bbe29c97454b9bf6b20fd7f9e678 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:46:39.835319 11069 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffa7bbe29c97454b9bf6b20fd7f9e678 [term 1 LEADER]: Becoming Leader. State: Replica: ffa7bbe29c97454b9bf6b20fd7f9e678, State: Running, Role: LEADER
I20260926 16:46:39.835657 11069 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffa7bbe29c97454b9bf6b20fd7f9e678 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa7bbe29c97454b9bf6b20fd7f9e678" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 44739 } }
I20260926 16:46:39.836148 11059 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffa7bbe29c97454b9bf6b20fd7f9e678 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:46:39.837291 11071 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffa7bbe29c97454b9bf6b20fd7f9e678 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffa7bbe29c97454b9bf6b20fd7f9e678. Latest consensus state: current_term: 1 leader_uuid: "ffa7bbe29c97454b9bf6b20fd7f9e678" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa7bbe29c97454b9bf6b20fd7f9e678" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 44739 } } }
I20260926 16:46:39.837426 11071 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffa7bbe29c97454b9bf6b20fd7f9e678 [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:39.837839 11078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:46:39.837805 11070 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffa7bbe29c97454b9bf6b20fd7f9e678 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ffa7bbe29c97454b9bf6b20fd7f9e678" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa7bbe29c97454b9bf6b20fd7f9e678" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 44739 } } }
I20260926 16:46:39.837949 11070 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffa7bbe29c97454b9bf6b20fd7f9e678 [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:39.841145 11078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:46:39.846310 11078 catalog_manager.cc:1347] Generated new cluster ID: 7545901a4e1f41f48f2cdc4187dfaa75
I20260926 16:46:39.846390 11078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:46:39.871419 11078 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:46:39.871802 11078 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:46:39.876415 11078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ffa7bbe29c97454b9bf6b20fd7f9e678: Generated new TSK 0
I20260926 16:46:39.877254 11078 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:46:39.880818 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 10991
2026-09-26T16:46:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1405 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440924937351-21553-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:46:39 dist-test-slave-n1hw krb5kdc[11090](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:46:39 dist-test-slave-n1hw krb5kdc[11090](info): set up 2 sockets
Sep 26 16:46:39 dist-test-slave-n1hw krb5kdc[11090](info): commencing operation
krb5kdc: starting...
W20260926 16:46:40.928166 21553 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Sep 26 16:46:41 dist-test-slave-n1hw krb5kdc[11090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:46:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:46:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
I20260926 16:46:41.049012 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:45927
--webserver_interface=127.21.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:37649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:45927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:46:41.163755 11103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:41.164009 11103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:41.164089 11103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:41.168246 11103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:46:41.168301 11103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:41.168322 11103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:41.168365 11103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:46:41.168395 11103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:46:41.173537 11103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:37649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:45927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:45927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:46:41.174587 11103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:41.175649 11103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:41.183830 11109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:41.183821 11108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:41.183821 11111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:41.184173 11103 server_base.cc:1034] running on GCE node
I20260926 16:46:41.184713 11103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:41.185842 11103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:41.187021 11103 hybrid_clock.cc:648] HybridClock initialized: now 1790441201187002 us; error 35 us; skew 500 ppm
Sep 26 16:46:41 dist-test-slave-n1hw krb5kdc[11090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441201, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:46:41.192847 11103 init.cc:377] Logged in from keytab as kudu/127.21.12.126@KRBTEST.COM (short username kudu)
I20260926 16:46:41.193831 11103 webserver.cc:460] Webserver started at http://127.21.12.126:38439/ using document root <none> and password file <none>
I20260926 16:46:41.194386 11103 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:41.194475 11103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:41.194702 11103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:46:41.196689 11103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "510e85f9d3664e77af3bd6c470bca3cc"
format_stamp: "Formatted at 2026-09-26 16:46:41 on dist-test-slave-n1hw"
server_key: "25de18d2380500b7fbb403409dd1328c"
server_key_iv: "bb17f6b33065bdf32d0507f160ce5c54"
server_key_version: "encryptionkey@0"
I20260926 16:46:41.197247 11103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "510e85f9d3664e77af3bd6c470bca3cc"
format_stamp: "Formatted at 2026-09-26 16:46:41 on dist-test-slave-n1hw"
server_key: "25de18d2380500b7fbb403409dd1328c"
server_key_iv: "bb17f6b33065bdf32d0507f160ce5c54"
server_key_version: "encryptionkey@0"
I20260926 16:46:41.200928 11103 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 16:46:41.203498 11118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:41.204834 11103 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 16:46:41.204972 11103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "510e85f9d3664e77af3bd6c470bca3cc"
format_stamp: "Formatted at 2026-09-26 16:46:41 on dist-test-slave-n1hw"
server_key: "25de18d2380500b7fbb403409dd1328c"
server_key_iv: "bb17f6b33065bdf32d0507f160ce5c54"
server_key_version: "encryptionkey@0"
I20260926 16:46:41.205083 11103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:41.231756 11103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:41.235307 11103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:41.235510 11103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:41.242538 11103 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.126:45927
I20260926 16:46:41.242558 11170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.126:45927 every 8 connection(s)
I20260926 16:46:41.243460 11103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:46:41.245116 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 11103
I20260926 16:46:41.245247 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:46:41.245577 21553 external_mini_cluster.cc:1428] Setting key 0ff432f8122f2a9dd19e296ab7fb18a6
I20260926 16:46:41.246757 11171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:41.253268 11171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 510e85f9d3664e77af3bd6c470bca3cc: Bootstrap starting.
Sep 26 16:46:41 dist-test-slave-n1hw krb5kdc[11090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:46:41.256067 11171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 510e85f9d3664e77af3bd6c470bca3cc: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:41.256778 11171 log.cc:824] T 00000000000000000000000000000000 P 510e85f9d3664e77af3bd6c470bca3cc: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:41.259084 11171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 510e85f9d3664e77af3bd6c470bca3cc: No bootstrap required, opened a new log
I20260926 16:46:41.260704 11174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:41.247113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47871 (local address 127.21.12.126:45927)
0926 16:46:41.247501 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:41.247522 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:41.247569 (+ 47us) server_negotiation.cc:407] Connection header received
0926 16:46:41.248219 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:41.248236 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:41.248570 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:41.248853 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:41.249853 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:41.250823 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:41.252026 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:41.252489 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:41.252604 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:41.255551 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:41.255556 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:41.255568 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:41.255595 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:41.257635 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:41.258167 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:41.258172 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:41.258177 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:41.258303 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:41.258653 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:41.258655 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:41.258656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:41.259106 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:41.259347 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:41.259656 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 16:46:41.259965 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":161,"threads_started":1}
I20260926 16:46:41.263159 11171 raft_consensus.cc:348] T 00000000000000000000000000000000 P 510e85f9d3664e77af3bd6c470bca3cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510e85f9d3664e77af3bd6c470bca3cc" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 45927 } }
I20260926 16:46:41.263453 11171 raft_consensus.cc:374] T 00000000000000000000000000000000 P 510e85f9d3664e77af3bd6c470bca3cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:46:41.263518 11171 raft_consensus.cc:729] T 00000000000000000000000000000000 P 510e85f9d3664e77af3bd6c470bca3cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 510e85f9d3664e77af3bd6c470bca3cc, State: Initialized, Role: FOLLOWER
I20260926 16:46:41.263764 21553 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:46:41.264000 11171 consensus_queue.cc:260] T 00000000000000000000000000000000 P 510e85f9d3664e77af3bd6c470bca3cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510e85f9d3664e77af3bd6c470bca3cc" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 45927 } }
I20260926 16:46:41.264153 11171 raft_consensus.cc:388] T 00000000000000000000000000000000 P 510e85f9d3664e77af3bd6c470bca3cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:46:41.264238 11171 raft_consensus.cc:482] T 00000000000000000000000000000000 P 510e85f9d3664e77af3bd6c470bca3cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:46:41.264334 11171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 510e85f9d3664e77af3bd6c470bca3cc [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:41.265372 11171 raft_consensus.cc:504] T 00000000000000000000000000000000 P 510e85f9d3664e77af3bd6c470bca3cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510e85f9d3664e77af3bd6c470bca3cc" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 45927 } }
I20260926 16:46:41.265739 11171 leader_election.cc:302] T 00000000000000000000000000000000 P 510e85f9d3664e77af3bd6c470bca3cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 510e85f9d3664e77af3bd6c470bca3cc; no voters:
I20260926 16:46:41.266100 11171 leader_election.cc:288] T 00000000000000000000000000000000 P 510e85f9d3664e77af3bd6c470bca3cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:46:41.266469 11182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 510e85f9d3664e77af3bd6c470bca3cc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:46:41.266685 11182 raft_consensus.cc:686] T 00000000000000000000000000000000 P 510e85f9d3664e77af3bd6c470bca3cc [term 1 LEADER]: Becoming Leader. State: Replica: 510e85f9d3664e77af3bd6c470bca3cc, State: Running, Role: LEADER
I20260926 16:46:41.267007 11182 consensus_queue.cc:237] T 00000000000000000000000000000000 P 510e85f9d3664e77af3bd6c470bca3cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510e85f9d3664e77af3bd6c470bca3cc" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 45927 } }
I20260926 16:46:41.267400 11171 sys_catalog.cc:564] T 00000000000000000000000000000000 P 510e85f9d3664e77af3bd6c470bca3cc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:46:41.268224 11182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 510e85f9d3664e77af3bd6c470bca3cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 510e85f9d3664e77af3bd6c470bca3cc. Latest consensus state: current_term: 1 leader_uuid: "510e85f9d3664e77af3bd6c470bca3cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510e85f9d3664e77af3bd6c470bca3cc" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 45927 } } }
I20260926 16:46:41.268384 11182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 510e85f9d3664e77af3bd6c470bca3cc [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:41.269172 11183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 510e85f9d3664e77af3bd6c470bca3cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "510e85f9d3664e77af3bd6c470bca3cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510e85f9d3664e77af3bd6c470bca3cc" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 45927 } } }
I20260926 16:46:41.269313 11183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 510e85f9d3664e77af3bd6c470bca3cc [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:41.269685 11189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:46:41.272953 11189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:46:41.274724 11174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:41.265928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37301 (local address 127.21.12.126:45927)
0926 16:46:41.266041 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:41.266045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:41.266174 (+ 129us) server_negotiation.cc:407] Connection header received
0926 16:46:41.266337 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:41.266340 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:41.266383 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:41.266473 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:41.267239 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:41.267850 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:41.269099 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:41.269471 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:41.269587 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:41.271298 (+ 1711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:41.271301 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:41.271304 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:41.271327 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:41.272845 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:41.273358 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:41.273361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:41.273363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:41.273472 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:41.273821 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:41.273824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:41.273826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:41.274092 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:41.274228 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:41.274450 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 16:46:41.274568 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 16:46:41.278489 11189 catalog_manager.cc:1347] Generated new cluster ID: 9a94478b99f34dceb1e64a50f26df57f
I20260926 16:46:41.278553 11189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:46:41.291004 11189 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:46:41.291889 11189 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:46:41.297222 11189 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 510e85f9d3664e77af3bd6c470bca3cc: Generated new TSK 0
I20260926 16:46:41.297951 11189 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:46:41.301602 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 11103
2026-09-26T16:46:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1420 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440924937351-21553-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:46:41 dist-test-slave-n1hw krb5kdc[11202](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:46:41 dist-test-slave-n1hw krb5kdc[11202](info): set up 2 sockets
Sep 26 16:46:41 dist-test-slave-n1hw krb5kdc[11202](info): commencing operation
krb5kdc: starting...
W20260926 16:46:42.348260 21553 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Sep 26 16:46:42 dist-test-slave-n1hw krb5kdc[11202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:46:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:46:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
I20260926 16:46:42.468423 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:40339
--webserver_interface=127.21.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:42287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:40339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:46:42.577797 11216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:42.578049 11216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:42.578089 11216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:42.582026 11216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:46:42.582078 11216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:42.582098 11216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:42.582114 11216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:46:42.582129 11216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:46:42.587312 11216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:42287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:40339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:40339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:46:42.588320 11216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:42.589354 11216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:42.597566 11224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:42.597615 11216 server_base.cc:1034] running on GCE node
W20260926 16:46:42.597577 11222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:42.597589 11221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:42.598475 11216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:42.599581 11216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:42.600780 11216 hybrid_clock.cc:648] HybridClock initialized: now 1790441202600753 us; error 37 us; skew 500 ppm
Sep 26 16:46:42 dist-test-slave-n1hw krb5kdc[11202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441202, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:46:42.603534 11216 init.cc:377] Logged in from keytab as kudu/127.21.12.126@KRBTEST.COM (short username kudu)
I20260926 16:46:42.604413 11216 webserver.cc:460] Webserver started at http://127.21.12.126:45343/ using document root <none> and password file <none>
I20260926 16:46:42.604945 11216 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:42.605031 11216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:42.605252 11216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:46:42.607244 11216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "db19bc29b8af4650b8669c943389858d"
format_stamp: "Formatted at 2026-09-26 16:46:42 on dist-test-slave-n1hw"
server_key: "b0d779da1d72fc3d88c792cac8e81f7a"
server_key_iv: "e9e73ce9e9b563a4164f96786b0c8727"
server_key_version: "encryptionkey@0"
I20260926 16:46:42.607801 11216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "db19bc29b8af4650b8669c943389858d"
format_stamp: "Formatted at 2026-09-26 16:46:42 on dist-test-slave-n1hw"
server_key: "b0d779da1d72fc3d88c792cac8e81f7a"
server_key_iv: "e9e73ce9e9b563a4164f96786b0c8727"
server_key_version: "encryptionkey@0"
I20260926 16:46:42.611325 11216 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260926 16:46:42.613651 11231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:42.614806 11216 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:46:42.614949 11216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "db19bc29b8af4650b8669c943389858d"
format_stamp: "Formatted at 2026-09-26 16:46:42 on dist-test-slave-n1hw"
server_key: "b0d779da1d72fc3d88c792cac8e81f7a"
server_key_iv: "e9e73ce9e9b563a4164f96786b0c8727"
server_key_version: "encryptionkey@0"
I20260926 16:46:42.615052 11216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:42.631698 11216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:42.634752 11216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:42.634945 11216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:42.641645 11216 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.126:40339
I20260926 16:46:42.641671 11283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.126:40339 every 8 connection(s)
I20260926 16:46:42.642575 11216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:46:42.644200 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 11216
I20260926 16:46:42.644331 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:46:42.644629 21553 external_mini_cluster.cc:1428] Setting key 9afd53f03758d617a2edb8e0e2c23550
I20260926 16:46:42.645464 11284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:42.650836 11284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db19bc29b8af4650b8669c943389858d: Bootstrap starting.
I20260926 16:46:42.652959 11284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db19bc29b8af4650b8669c943389858d: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:42.653033 11287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:42.645992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48379 (local address 127.21.12.126:40339)
0926 16:46:42.646414 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:42.646428 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:42.646462 (+ 34us) server_negotiation.cc:407] Connection header received
0926 16:46:42.647072 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:42.647088 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:42.647412 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:42.647722 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:42.648098 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:42.649050 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:42.650040 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:42.650499 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:42.650670 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:42.651521 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:42.651526 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:42.651537 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:46:42.651889 (+ 352us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:42.651941 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:46:42.651963 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:42.652066 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:42.652198 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 16:46:42.652395 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":134,"threads_started":1}
I20260926 16:46:42.653714 11284 log.cc:824] T 00000000000000000000000000000000 P db19bc29b8af4650b8669c943389858d: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:42.655747 21553 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:46:42.655849 11284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db19bc29b8af4650b8669c943389858d: No bootstrap required, opened a new log
I20260926 16:46:42.659766 11284 raft_consensus.cc:348] T 00000000000000000000000000000000 P db19bc29b8af4650b8669c943389858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db19bc29b8af4650b8669c943389858d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 40339 } }
I20260926 16:46:42.659996 11284 raft_consensus.cc:374] T 00000000000000000000000000000000 P db19bc29b8af4650b8669c943389858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:46:42.660060 11284 raft_consensus.cc:729] T 00000000000000000000000000000000 P db19bc29b8af4650b8669c943389858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db19bc29b8af4650b8669c943389858d, State: Initialized, Role: FOLLOWER
I20260926 16:46:42.660531 11284 consensus_queue.cc:260] T 00000000000000000000000000000000 P db19bc29b8af4650b8669c943389858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db19bc29b8af4650b8669c943389858d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 40339 } }
I20260926 16:46:42.660701 11284 raft_consensus.cc:388] T 00000000000000000000000000000000 P db19bc29b8af4650b8669c943389858d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:46:42.660772 11284 raft_consensus.cc:482] T 00000000000000000000000000000000 P db19bc29b8af4650b8669c943389858d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:46:42.660867 11284 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db19bc29b8af4650b8669c943389858d [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:42.661449 11287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:42.657623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33463 (local address 127.21.12.126:40339)
0926 16:46:42.657730 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:42.657733 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:42.658012 (+ 279us) server_negotiation.cc:407] Connection header received
0926 16:46:42.658148 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:42.658151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:42.658211 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:42.658286 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:42.658531 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:42.659027 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:42.659799 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:42.660080 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:42.660204 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:42.661032 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:42.661035 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:42.661038 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:46:42.661051 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:42.661077 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:46:42.661088 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:42.661163 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:42.661257 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 16:46:42.661317 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 16:46:42.661953 11284 raft_consensus.cc:504] T 00000000000000000000000000000000 P db19bc29b8af4650b8669c943389858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db19bc29b8af4650b8669c943389858d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 40339 } }
I20260926 16:46:42.662290 11284 leader_election.cc:302] T 00000000000000000000000000000000 P db19bc29b8af4650b8669c943389858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db19bc29b8af4650b8669c943389858d; no voters:
I20260926 16:46:42.662606 11284 leader_election.cc:288] T 00000000000000000000000000000000 P db19bc29b8af4650b8669c943389858d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:46:42.662710 11294 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db19bc29b8af4650b8669c943389858d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:46:42.662963 11294 raft_consensus.cc:686] T 00000000000000000000000000000000 P db19bc29b8af4650b8669c943389858d [term 1 LEADER]: Becoming Leader. State: Replica: db19bc29b8af4650b8669c943389858d, State: Running, Role: LEADER
I20260926 16:46:42.663301 11294 consensus_queue.cc:237] T 00000000000000000000000000000000 P db19bc29b8af4650b8669c943389858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db19bc29b8af4650b8669c943389858d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 40339 } }
I20260926 16:46:42.663800 11284 sys_catalog.cc:564] T 00000000000000000000000000000000 P db19bc29b8af4650b8669c943389858d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:46:42.664821 11296 sys_catalog.cc:455] T 00000000000000000000000000000000 P db19bc29b8af4650b8669c943389858d [sys.catalog]: SysCatalogTable state changed. Reason: New leader db19bc29b8af4650b8669c943389858d. Latest consensus state: current_term: 1 leader_uuid: "db19bc29b8af4650b8669c943389858d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db19bc29b8af4650b8669c943389858d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 40339 } } }
I20260926 16:46:42.664937 11296 sys_catalog.cc:458] T 00000000000000000000000000000000 P db19bc29b8af4650b8669c943389858d [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:42.665349 11304 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:46:42.665325 11295 sys_catalog.cc:455] T 00000000000000000000000000000000 P db19bc29b8af4650b8669c943389858d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db19bc29b8af4650b8669c943389858d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db19bc29b8af4650b8669c943389858d" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 40339 } } }
I20260926 16:46:42.665988 11295 sys_catalog.cc:458] T 00000000000000000000000000000000 P db19bc29b8af4650b8669c943389858d [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:42.668483 11304 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:46:42.673502 11304 catalog_manager.cc:1347] Generated new cluster ID: 038095dc7d4344949673d731a1128ab7
I20260926 16:46:42.673564 11304 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:46:42.681396 11304 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:46:42.682328 11304 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:46:42.690330 11304 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db19bc29b8af4650b8669c943389858d: Generated new TSK 0
I20260926 16:46:42.690992 11304 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:46:42.697180 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 11216
2026-09-26T16:46:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1394 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440924937351-21553-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:46:42 dist-test-slave-n1hw krb5kdc[11316](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:46:42 dist-test-slave-n1hw krb5kdc[11316](info): set up 2 sockets
Sep 26 16:46:42 dist-test-slave-n1hw krb5kdc[11316](info): commencing operation
krb5kdc: starting...
W20260926 16:46:43.732221 21553 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Sep 26 16:46:43 dist-test-slave-n1hw krb5kdc[11316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:46:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:46:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
I20260926 16:46:43.857074 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:33583
--webserver_interface=127.21.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:42159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:33583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:46:43.963937 11329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:43.964243 11329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:43.964318 11329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:43.968370 11329 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:46:43.968495 11329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:43.968533 11329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:43.968583 11329 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:46:43.968616 11329 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:46:43.973980 11329 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:42159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:33583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:33583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:46:43.975064 11329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:43.976099 11329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:43.984648 11337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:43.984659 11335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:43.984654 11334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:43.984818 11329 server_base.cc:1034] running on GCE node
I20260926 16:46:43.985587 11329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:43.986748 11329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:43.987941 11329 hybrid_clock.cc:648] HybridClock initialized: now 1790441203987910 us; error 47 us; skew 500 ppm
Sep 26 16:46:43 dist-test-slave-n1hw krb5kdc[11316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441203, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:46:43.990870 11329 init.cc:377] Logged in from keytab as kudu/127.21.12.126@KRBTEST.COM (short username kudu)
I20260926 16:46:43.991737 11329 webserver.cc:460] Webserver started at http://127.21.12.126:35763/ using document root <none> and password file <none>
I20260926 16:46:43.992381 11329 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:43.992475 11329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:43.992702 11329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:46:43.994777 11329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "80087f94a5794ab5b55855a9adc5ebfa"
format_stamp: "Formatted at 2026-09-26 16:46:43 on dist-test-slave-n1hw"
server_key: "2091cab61be254df7c4fd69f769b0ddf"
server_key_iv: "875a9c42fa431655563327eb6aff5d47"
server_key_version: "encryptionkey@0"
I20260926 16:46:43.995348 11329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "80087f94a5794ab5b55855a9adc5ebfa"
format_stamp: "Formatted at 2026-09-26 16:46:43 on dist-test-slave-n1hw"
server_key: "2091cab61be254df7c4fd69f769b0ddf"
server_key_iv: "875a9c42fa431655563327eb6aff5d47"
server_key_version: "encryptionkey@0"
I20260926 16:46:43.999060 11329 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:46:44.001530 11344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:44.002948 11329 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:46:44.003113 11329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "80087f94a5794ab5b55855a9adc5ebfa"
format_stamp: "Formatted at 2026-09-26 16:46:43 on dist-test-slave-n1hw"
server_key: "2091cab61be254df7c4fd69f769b0ddf"
server_key_iv: "875a9c42fa431655563327eb6aff5d47"
server_key_version: "encryptionkey@0"
I20260926 16:46:44.003232 11329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:44.007858 11329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:44.011195 11329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:44.011387 11329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:44.018414 11329 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.126:33583
I20260926 16:46:44.018496 11396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.126:33583 every 8 connection(s)
I20260926 16:46:44.019343 11329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:46:44.022048 11397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:44.022687 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 11329
I20260926 16:46:44.022817 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:46:44.023109 21553 external_mini_cluster.cc:1428] Setting key 0abbe09c31c87ef55665fcb55cb127f5
I20260926 16:46:44.028534 11400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:44.024468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41021 (local address 127.21.12.126:33583)
0926 16:46:44.024904 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:44.024922 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:44.024966 (+ 44us) server_negotiation.cc:407] Connection header received
0926 16:46:44.025801 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:44.025824 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:44.026162 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:44.026518 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:44.027206 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:44.027212 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:44.027227 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:46:44.027238 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:44.027298 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:46:44.027322 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:44.027467 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:44.027540 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 16:46:44.027688 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":100,"threads_started":1}
I20260926 16:46:44.028728 11397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80087f94a5794ab5b55855a9adc5ebfa: Bootstrap starting.
I20260926 16:46:44.030959 11397 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80087f94a5794ab5b55855a9adc5ebfa: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:44.031219 21553 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:46:44.031852 11397 log.cc:824] T 00000000000000000000000000000000 P 80087f94a5794ab5b55855a9adc5ebfa: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:44.033859 11397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80087f94a5794ab5b55855a9adc5ebfa: No bootstrap required, opened a new log
I20260926 16:46:44.035017 11400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:44.033476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48983 (local address 127.21.12.126:33583)
0926 16:46:44.033597 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:44.033601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:44.033744 (+ 143us) server_negotiation.cc:407] Connection header received
0926 16:46:44.033910 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:44.033913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:44.033946 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:44.034022 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:44.034683 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:44.034687 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:44.034689 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:46:44.034701 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:44.034727 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:46:44.034739 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:44.034832 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:44.034877 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 16:46:44.034910 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 16:46:44.037711 11397 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80087f94a5794ab5b55855a9adc5ebfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80087f94a5794ab5b55855a9adc5ebfa" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 33583 } }
I20260926 16:46:44.037988 11397 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80087f94a5794ab5b55855a9adc5ebfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:46:44.038064 11397 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80087f94a5794ab5b55855a9adc5ebfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80087f94a5794ab5b55855a9adc5ebfa, State: Initialized, Role: FOLLOWER
I20260926 16:46:44.038512 11397 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80087f94a5794ab5b55855a9adc5ebfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80087f94a5794ab5b55855a9adc5ebfa" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 33583 } }
I20260926 16:46:44.038653 11397 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80087f94a5794ab5b55855a9adc5ebfa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:46:44.038720 11397 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80087f94a5794ab5b55855a9adc5ebfa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:46:44.038817 11397 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80087f94a5794ab5b55855a9adc5ebfa [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:44.039683 11397 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80087f94a5794ab5b55855a9adc5ebfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80087f94a5794ab5b55855a9adc5ebfa" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 33583 } }
I20260926 16:46:44.040030 11397 leader_election.cc:302] T 00000000000000000000000000000000 P 80087f94a5794ab5b55855a9adc5ebfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80087f94a5794ab5b55855a9adc5ebfa; no voters:
I20260926 16:46:44.040325 11397 leader_election.cc:288] T 00000000000000000000000000000000 P 80087f94a5794ab5b55855a9adc5ebfa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:46:44.040458 11407 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80087f94a5794ab5b55855a9adc5ebfa [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:46:44.040691 11407 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80087f94a5794ab5b55855a9adc5ebfa [term 1 LEADER]: Becoming Leader. State: Replica: 80087f94a5794ab5b55855a9adc5ebfa, State: Running, Role: LEADER
I20260926 16:46:44.041074 11407 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80087f94a5794ab5b55855a9adc5ebfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80087f94a5794ab5b55855a9adc5ebfa" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 33583 } }
I20260926 16:46:44.041564 11397 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80087f94a5794ab5b55855a9adc5ebfa [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:46:44.042919 11409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80087f94a5794ab5b55855a9adc5ebfa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80087f94a5794ab5b55855a9adc5ebfa. Latest consensus state: current_term: 1 leader_uuid: "80087f94a5794ab5b55855a9adc5ebfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80087f94a5794ab5b55855a9adc5ebfa" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 33583 } } }
I20260926 16:46:44.042968 11408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80087f94a5794ab5b55855a9adc5ebfa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80087f94a5794ab5b55855a9adc5ebfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80087f94a5794ab5b55855a9adc5ebfa" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 33583 } } }
I20260926 16:46:44.043045 11409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80087f94a5794ab5b55855a9adc5ebfa [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:44.043077 11408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80087f94a5794ab5b55855a9adc5ebfa [sys.catalog]: This master's current role is: LEADER
W20260926 16:46:44.046368 11422 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 80087f94a5794ab5b55855a9adc5ebfa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 16:46:44.046448 11422 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 16:46:44.046554 11417 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:46:44.047422 11417 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:46:44.053089 11417 catalog_manager.cc:1347] Generated new cluster ID: bf6fde54cf7349a895a2367ac16e91f9
I20260926 16:46:44.053169 11417 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:46:44.062793 11417 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:46:44.063194 11417 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:46:44.072315 11417 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80087f94a5794ab5b55855a9adc5ebfa: Generated new TSK 0
I20260926 16:46:44.072981 11417 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:46:44.085079 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 11329
2026-09-26T16:46:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1388 ms)
[----------] 18 tests from AuthTokenIssuingTest (25507 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:46:44 dist-test-slave-n1hw krb5kdc[11428](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:46:44 dist-test-slave-n1hw krb5kdc[11428](info): set up 2 sockets
Sep 26 16:46:44 dist-test-slave-n1hw krb5kdc[11428](info): commencing operation
krb5kdc: starting...
W20260926 16:46:45.128053 21553 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Sep 26 16:46:45 dist-test-slave-n1hw krb5kdc[11428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:46:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:46:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
I20260926 16:46:45.252463 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:46369
--webserver_interface=127.21.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:32873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:46369,127.21.12.125:43535,127.21.12.124:44661
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:46:45.362589 11442 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 16:46:45.362870 11442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:45.362895 11442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:45.362931 11442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:45.366775 11442 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:46:45.366822 11442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:45.366846 11442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:45.366863 11442 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:46:45.366878 11442 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:46:45.372033 11442 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:32873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:46369,127.21.12.125:43535,127.21.12.124:44661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:46369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:46:45.372980 11442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:45.373946 11442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:45.382195 11447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:45.382288 11442 server_base.cc:1034] running on GCE node
W20260926 16:46:45.382215 11448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:45.382215 11450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:45.382957 11442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:45.384054 11442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:45.385254 11442 hybrid_clock.cc:648] HybridClock initialized: now 1790441205385234 us; error 31 us; skew 500 ppm
Sep 26 16:46:45 dist-test-slave-n1hw krb5kdc[11428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441205, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:46:45.388075 11442 init.cc:377] Logged in from keytab as kudu/127.21.12.126@KRBTEST.COM (short username kudu)
I20260926 16:46:45.388862 11442 webserver.cc:460] Webserver started at http://127.21.12.126:33329/ using document root <none> and password file <none>
I20260926 16:46:45.389374 11442 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:45.389432 11442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:45.389590 11442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:46:45.391623 11442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "6e6c4b13b244400684a8cd8a3b346d21"
format_stamp: "Formatted at 2026-09-26 16:46:45 on dist-test-slave-n1hw"
server_key: "8c893a984fd1a56d15b4b1f1245cf438"
server_key_iv: "dc222b8cf29f8cdc79e062ad86b6645c"
server_key_version: "encryptionkey@0"
I20260926 16:46:45.392138 11442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "6e6c4b13b244400684a8cd8a3b346d21"
format_stamp: "Formatted at 2026-09-26 16:46:45 on dist-test-slave-n1hw"
server_key: "8c893a984fd1a56d15b4b1f1245cf438"
server_key_iv: "dc222b8cf29f8cdc79e062ad86b6645c"
server_key_version: "encryptionkey@0"
I20260926 16:46:45.395733 11442 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 16:46:45.398144 11457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:45.399389 11442 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:46:45.399535 11442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "6e6c4b13b244400684a8cd8a3b346d21"
format_stamp: "Formatted at 2026-09-26 16:46:45 on dist-test-slave-n1hw"
server_key: "8c893a984fd1a56d15b4b1f1245cf438"
server_key_iv: "dc222b8cf29f8cdc79e062ad86b6645c"
server_key_version: "encryptionkey@0"
I20260926 16:46:45.399634 11442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:45.411546 11442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:45.414418 11442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:45.414621 11442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:45.421741 11442 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.126:46369
I20260926 16:46:45.422052 11509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.126:46369 every 8 connection(s)
I20260926 16:46:45.422976 11442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:46:45.425832 11510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:45.428418 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 11442
I20260926 16:46:45.428557 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:46:45.428872 21553 external_mini_cluster.cc:1428] Setting key a6a310b265fb8f473f9e9bdb0e76de12
I20260926 16:46:45.429865 11510 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46369 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.12.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.125@KRBTEST.COM" created.
Sep 26 16:46:45 dist-test-slave-n1hw krb5kdc[11428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441205, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:46:45.446419 11514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:45.432156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52704 (local address 127.21.12.126:46369)
0926 16:46:45.432709 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:45.432724 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:45.432791 (+ 67us) server_negotiation.cc:407] Connection header received
0926 16:46:45.433867 (+ 1076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:45.433877 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:45.434137 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:45.434370 (+ 233us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:45.435149 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:45.436639 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:45.437685 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:45.438153 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:45.438294 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:45.441426 (+ 3132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:45.441431 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:45.441441 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:45.441463 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:45.443179 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:45.443679 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:45.443687 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:45.443693 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:45.443791 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:45.444145 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:45.444150 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:45.444152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:45.444581 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:45.444715 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:45.445079 (+ 364us) server_negotiation.cc:299] Negotiation successful
0926 16:46:45.445354 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":424,"thread_start_us":138,"threads_started":1}
I20260926 16:46:45.446419 11513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:45.432155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:46369 (local address 127.0.0.1:52704)
0926 16:46:45.432623 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:45.432675 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:45.433495 (+ 820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:45.434367 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:45.434372 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:45.434697 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:46:45.434983 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:45.434996 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:45.436768 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:45.436770 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:45.437567 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:45.437576 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:45.438307 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:45.438310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:45.438416 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:45.438998 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:46:45.439014 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:46:45.441261 (+ 2247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:46:45.443314 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:45.443325 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:45.443334 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:45.443557 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:45.443936 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:45.443939 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:45.443940 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:45.444049 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:45.444700 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:46:45.444706 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:46:45.444834 (+ 128us) client_negotiation.cc:769] Sending connection context
0926 16:46:45.445006 (+ 172us) client_negotiation.cc:240] Negotiation successful
0926 16:46:45.445403 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":111,"thread_start_us":138,"threads_started":1}
I20260926 16:46:45.448755 11510 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.125" port: 43535 } has no permanent_uuid. Determining permanent_uuid...
I20260926 16:46:45.449574 11513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:45.449308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.125:43535 (local address 127.0.0.1:47196)
0926 16:46:45.449444 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:45.449514 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.125:43535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260926 16:46:45.449820 11458 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.12.125:43535: connect: Connection refused (error 111)
I20260926 16:46:45.450654 11513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:45.450450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.125:43535 (local address 127.0.0.1:47200)
0926 16:46:45.450556 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:45.450601 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.125:43535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 16:46:45.450922 11510 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.125:43535: Network error: Client connection negotiation failed: client connection to 127.21.12.125:43535: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.125.keytab.
Entry for principal kudu/127.21.12.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.125.keytab.
I20260926 16:46:45.457108 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.125:43535
--webserver_interface=127.21.12.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.125.keytab
--principal=kudu/127.21.12.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:32873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:46369,127.21.12.125:43535,127.21.12.124:44661
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:46:45.500209 11510 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.125" port: 43535 } attempt: 1
I20260926 16:46:45.501137 11513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:45.500798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.125:43535 (local address 127.0.0.1:47210)
0926 16:46:45.500971 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:45.501044 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.125:43535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260926 16:46:45.501997 11513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:45.501728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.125:43535 (local address 127.0.0.1:47216)
0926 16:46:45.501867 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:45.501929 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.125:43535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260926 16:46:45.502163 11510 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.125:43535: Network error: Client connection negotiation failed: client connection to 127.21.12.125:43535: connect: Connection refused (error 111)
W20260926 16:46:45.567502 11517 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 16:46:45.567770 11517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:45.567793 11517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:45.567831 11517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 16:46:45.570330 11510 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.125" port: 43535 } attempt: 2
I20260926 16:46:45.571269 11513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:45.570938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.125:43535 (local address 127.0.0.1:47222)
0926 16:46:45.571110 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:45.571181 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.125:43535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 16:46:45.571996 11513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:45.571786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.125:43535 (local address 127.0.0.1:47228)
0926 16:46:45.571886 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:45.571935 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.125:43535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 16:46:45.572119 11510 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.125:43535: Network error: Client connection negotiation failed: client connection to 127.21.12.125:43535: connect: Connection refused (error 111)
W20260926 16:46:45.572499 11517 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:46:45.572577 11517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:45.572621 11517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:45.572657 11517 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:46:45.572691 11517 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:46:45.578804 11517 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:32873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:46369,127.21.12.125:43535,127.21.12.124:44661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.125.keytab
--principal=kudu/127.21.12.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.125:43535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.12.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:46:45.579895 11517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:45.580924 11517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:45.589465 11525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:45.589484 11523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:45.589578 11517 server_base.cc:1034] running on GCE node
W20260926 16:46:45.589473 11522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:45.591091 11517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:45.592163 11517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:45.593376 11517 hybrid_clock.cc:648] HybridClock initialized: now 1790441205593346 us; error 42 us; skew 500 ppm
Sep 26 16:46:45 dist-test-slave-n1hw krb5kdc[11428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441205, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:46:45.596207 11517 init.cc:377] Logged in from keytab as kudu/127.21.12.125@KRBTEST.COM (short username kudu)
I20260926 16:46:45.597038 11517 webserver.cc:460] Webserver started at http://127.21.12.125:33997/ using document root <none> and password file <none>
I20260926 16:46:45.597539 11517 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:45.597595 11517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:45.597806 11517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:46:45.599785 11517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-1/data/instance:
uuid: "e77bfa21c6384f5e932d7f58218ae7ca"
format_stamp: "Formatted at 2026-09-26 16:46:45 on dist-test-slave-n1hw"
server_key: "a41f4c3abb58a289f59cced56e1bd90a"
server_key_iv: "280cda1da18f1596a5ab32fcc5826478"
server_key_version: "encryptionkey@0"
I20260926 16:46:45.600277 11517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-1/wal/instance:
uuid: "e77bfa21c6384f5e932d7f58218ae7ca"
format_stamp: "Formatted at 2026-09-26 16:46:45 on dist-test-slave-n1hw"
server_key: "a41f4c3abb58a289f59cced56e1bd90a"
server_key_iv: "280cda1da18f1596a5ab32fcc5826478"
server_key_version: "encryptionkey@0"
I20260926 16:46:45.604067 11517 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 16:46:45.606446 11532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:45.607702 11517 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:46:45.607801 11517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-1/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-1/wal
uuid: "e77bfa21c6384f5e932d7f58218ae7ca"
format_stamp: "Formatted at 2026-09-26 16:46:45 on dist-test-slave-n1hw"
server_key: "a41f4c3abb58a289f59cced56e1bd90a"
server_key_iv: "280cda1da18f1596a5ab32fcc5826478"
server_key_version: "encryptionkey@0"
I20260926 16:46:45.607930 11517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:45.633126 11517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:45.636179 11517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:45.636329 11517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:45.643005 11517 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.125:43535
I20260926 16:46:45.643303 11585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.125:43535 every 8 connection(s)
I20260926 16:46:45.644025 11517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-1/data/info.pb
I20260926 16:46:45.646952 11586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:45.650532 11586 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46369 } has no permanent_uuid. Determining permanent_uuid...
I20260926 16:46:45.653690 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 11517
I20260926 16:46:45.653847 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-1/wal/instance
I20260926 16:46:45.654196 21553 external_mini_cluster.cc:1428] Setting key 8e356610917288a3dfb6e4ff4431f320
Sep 26 16:46:45 dist-test-slave-n1hw krb5kdc[11428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441205, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.125@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.12.124@KRBTEST.COM; defaulting to no policy
I20260926 16:46:45.663260 11510 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.125" port: 43535 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.124@KRBTEST.COM" created.
I20260926 16:46:45.666589 11514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:45.652345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52710 (local address 127.21.12.126:46369)
0926 16:46:45.652492 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:45.652497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:45.653412 (+ 915us) server_negotiation.cc:407] Connection header received
0926 16:46:45.654827 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:45.654830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:45.654886 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:45.654956 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:45.656537 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:45.657387 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:45.658756 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:45.659214 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:45.659352 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:45.662102 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:45.662107 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:45.662113 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:45.662143 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:45.664102 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:45.664692 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:45.664696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:45.664698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:45.664790 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:45.665205 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:45.665209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:45.665211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:45.665483 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:45.665583 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:45.666291 (+ 708us) server_negotiation.cc:299] Negotiation successful
0926 16:46:45.666421 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 16:46:45.667483 11588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:45.652750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:46369 (local address 127.0.0.1:52710)
0926 16:46:45.653258 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:45.653303 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:45.654565 (+ 1262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:45.655182 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:45.655195 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:45.655734 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:46:45.656378 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:45.656392 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:45.657523 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:45.657526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:45.658628 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:45.658638 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:45.659378 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:45.659381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:45.659527 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:45.660191 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:46:45.660209 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:46:45.661932 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:46:45.664237 (+ 2305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:45.664241 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:45.664252 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:45.664572 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:45.664892 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:45.664895 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:45.664897 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:45.665079 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:45.665644 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:46:45.665649 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:46:45.665989 (+ 340us) client_negotiation.cc:769] Sending connection context
0926 16:46:45.666189 (+ 200us) client_negotiation.cc:240] Negotiation successful
0926 16:46:45.666534 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":109,"threads_started":1}
I20260926 16:46:45.669431 11586 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.125" port: 43535 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:46:45 dist-test-slave-n1hw krb5kdc[11428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441205, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for kudu/127.21.12.125@KRBTEST.COM
Sep 26 16:46:45 dist-test-slave-n1hw krb5kdc[11428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441205, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.125@KRBTEST.COM for kudu/127.21.12.125@KRBTEST.COM
I20260926 16:46:45.674800 11513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:45.663790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.125:43535 (local address 127.0.0.1:47232)
0926 16:46:45.663894 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:45.663908 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:45.663982 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:45.664725 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:45.664727 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:45.664889 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:46:45.665049 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:45.665057 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:45.666480 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:45.666487 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:45.667400 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:45.667413 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:45.668249 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:45.668255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:45.668391 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:45.668933 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:46:45.668944 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:46:45.670534 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:46:45.672902 (+ 2368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:45.672905 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:45.672908 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:45.673167 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:45.673543 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:45.673548 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:45.673551 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:45.673645 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:45.674238 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:46:45.674241 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:46:45.674357 (+ 116us) client_negotiation.cc:769] Sending connection context
0926 16:46:45.674499 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 16:46:45.674635 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 16:46:45.674903 11590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:45.663915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47232 (local address 127.21.12.125:43535)
0926 16:46:45.664273 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:45.664285 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:45.664309 (+ 24us) server_negotiation.cc:407] Connection header received
0926 16:46:45.664353 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:45.664358 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:45.664570 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:45.664729 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:45.665194 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:45.666327 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:45.667574 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:45.668111 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:45.668631 (+ 520us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:45.670691 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:45.670701 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:45.670714 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:45.670738 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:45.672774 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:45.673314 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:45.673320 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:45.673325 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:45.673430 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:45.673786 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:45.673793 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:45.673796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:45.674108 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:45.674256 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:45.674601 (+ 345us) server_negotiation.cc:299] Negotiation successful
0926 16:46:45.674732 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":142,"threads_started":1}
I20260926 16:46:45.676829 11510 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.124" port: 44661 } has no permanent_uuid. Determining permanent_uuid...
I20260926 16:46:45.677510 11513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:45.677293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:44661 (local address 127.0.0.1:51642)
0926 16:46:45.677384 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:45.677440 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.124:44661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 16:46:45.678215 11513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:45.677929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:44661 (local address 127.0.0.1:51644)
0926 16:46:45.678087 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:45.678138 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.124:44661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260926 16:46:45.678362 11510 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.124:44661: Network error: Client connection negotiation failed: client connection to 127.21.12.124:44661: connect: Connection refused (error 111)
I20260926 16:46:45.678848 11588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:45.669955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.125:43535 (local address 127.0.0.1:47242)
0926 16:46:45.670063 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:45.670077 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:45.670148 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:45.670534 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:45.670537 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:45.670696 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:46:45.670769 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:45.670773 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:45.671858 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:45.671861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:45.672343 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:45.672351 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:45.672805 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:45.672808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:45.672917 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:45.673380 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:46:45.673392 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:46:45.674638 (+ 1246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:46:45.677281 (+ 2643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:45.677286 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:45.677290 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:45.677543 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:45.677915 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:45.677918 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:45.677920 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:45.678013 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:45.678455 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:46:45.678459 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:46:45.678561 (+ 102us) client_negotiation.cc:769] Sending connection context
0926 16:46:45.678649 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 16:46:45.678741 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":1,"spinlock_wait_cycles":44928}
I20260926 16:46:45.679061 11592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:45.670131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47242 (local address 127.21.12.125:43535)
0926 16:46:45.670359 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:45.670365 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:45.670375 (+ 10us) server_negotiation.cc:407] Connection header received
0926 16:46:45.670398 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:45.670400 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:45.670431 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:45.670525 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:45.670873 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:45.671568 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:45.672474 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:45.672704 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:45.672808 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:45.674791 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:45.674794 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:45.674797 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:45.674817 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:45.677169 (+ 2352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:45.677672 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:45.677675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:45.677677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:45.677802 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:45.678114 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:45.678119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:45.678120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:45.678328 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:45.678462 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:45.678711 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 16:46:45.678799 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":171,"thread_start_us":96,"threads_started":1}
I20260926 16:46:45.679652 11586 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.124" port: 44661 } has no permanent_uuid. Determining permanent_uuid...
I20260926 16:46:45.680255 11588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:45.680030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:44661 (local address 127.0.0.1:51656)
0926 16:46:45.680133 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:45.680206 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.124:44661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 16:46:45.680505 11534 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.12.124:44661: connect: Connection refused (error 111)
I20260926 16:46:45.681299 11588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:45.681094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:44661 (local address 127.0.0.1:51660)
0926 16:46:45.681215 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:45.681256 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.124:44661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 16:46:45.681438 11586 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.124:44661: Network error: Client connection negotiation failed: client connection to 127.21.12.124:44661: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.124.keytab.
Entry for principal kudu/127.21.12.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.124.keytab.
I20260926 16:46:45.684689 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.124:44661
--webserver_interface=127.21.12.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.124.keytab
--principal=kudu/127.21.12.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:32873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:46369,127.21.12.125:43535,127.21.12.124:44661
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:46:45.709581 11510 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.124" port: 44661 } attempt: 1
I20260926 16:46:45.710438 11513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:45.710134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:44661 (local address 127.0.0.1:51662)
0926 16:46:45.710284 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:45.710345 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.124:44661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 16:46:45.711233 11513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:45.711040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:44661 (local address 127.0.0.1:51670)
0926 16:46:45.711137 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:45.711181 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.124:44661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 16:46:45.711393 11510 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.124:44661: Network error: Client connection negotiation failed: client connection to 127.21.12.124:44661: connect: Connection refused (error 111)
I20260926 16:46:45.730710 11586 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.124" port: 44661 } attempt: 1
I20260926 16:46:45.731551 11588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:45.731226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:44661 (local address 127.0.0.1:51676)
0926 16:46:45.731379 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:45.731449 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.124:44661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 16:46:45.732270 11588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:45.732070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:44661 (local address 127.0.0.1:51692)
0926 16:46:45.732166 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:45.732210 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.124:44661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 16:46:45.732478 11586 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.124:44661: Network error: Client connection negotiation failed: client connection to 127.21.12.124:44661: connect: Connection refused (error 111)
I20260926 16:46:45.786576 11510 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.124" port: 44661 } attempt: 2
I20260926 16:46:45.787417 11513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:45.787082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:44661 (local address 127.0.0.1:51694)
0926 16:46:45.787263 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:45.787328 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.124:44661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 16:46:45.788300 11513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:45.788074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:44661 (local address 127.0.0.1:51700)
0926 16:46:45.788191 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:45.788243 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.124:44661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 16:46:45.788625 11510 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.124:44661: Network error: Client connection negotiation failed: client connection to 127.21.12.124:44661: connect: Connection refused (error 111)
W20260926 16:46:45.795513 11594 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 16:46:45.795763 11594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:45.795816 11594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:45.795882 11594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:45.799957 11594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:46:45.800006 11594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:45.800026 11594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:45.800042 11594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:46:45.800099 11594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:46:45.800637 11586 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.124" port: 44661 } attempt: 2
I20260926 16:46:45.801404 11588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:45.801124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:44661 (local address 127.0.0.1:51716)
0926 16:46:45.801266 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:45.801332 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.124:44661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 16:46:45.802311 11588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:45.802090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:44661 (local address 127.0.0.1:51728)
0926 16:46:45.802211 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:45.802260 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.124:44661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 16:46:45.802492 11586 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.124:44661: Network error: Client connection negotiation failed: client connection to 127.21.12.124:44661: connect: Connection refused (error 111)
I20260926 16:46:45.806116 11594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:32873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:46369,127.21.12.125:43535,127.21.12.124:44661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.124.keytab
--principal=kudu/127.21.12.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.124:44661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.12.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:46:45.807204 11594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:45.808210 11594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:45.816614 11602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:45.816640 11599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:45.816737 11594 server_base.cc:1034] running on GCE node
W20260926 16:46:45.816619 11600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:45.817477 11594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:45.818588 11594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:45.819744 11594 hybrid_clock.cc:648] HybridClock initialized: now 1790441205819726 us; error 30 us; skew 500 ppm
Sep 26 16:46:45 dist-test-slave-n1hw krb5kdc[11428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441205, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:46:45.822384 11594 init.cc:377] Logged in from keytab as kudu/127.21.12.124@KRBTEST.COM (short username kudu)
I20260926 16:46:45.823243 11594 webserver.cc:460] Webserver started at http://127.21.12.124:34469/ using document root <none> and password file <none>
I20260926 16:46:45.823740 11594 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:45.823796 11594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:45.823947 11594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:46:45.825872 11594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-2/data/instance:
uuid: "77a99ea0822f40c0ae7e9ec14affeb7d"
format_stamp: "Formatted at 2026-09-26 16:46:45 on dist-test-slave-n1hw"
server_key: "f4b42364e11ea27cca1d0e479c60a1d8"
server_key_iv: "767749fab6d2f7751fca787f214b25ac"
server_key_version: "encryptionkey@0"
I20260926 16:46:45.826361 11594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-2/wal/instance:
uuid: "77a99ea0822f40c0ae7e9ec14affeb7d"
format_stamp: "Formatted at 2026-09-26 16:46:45 on dist-test-slave-n1hw"
server_key: "f4b42364e11ea27cca1d0e479c60a1d8"
server_key_iv: "767749fab6d2f7751fca787f214b25ac"
server_key_version: "encryptionkey@0"
I20260926 16:46:45.829727 11594 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:46:45.832002 11609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:45.833140 11594 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:46:45.833262 11594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-2/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-2/wal
uuid: "77a99ea0822f40c0ae7e9ec14affeb7d"
format_stamp: "Formatted at 2026-09-26 16:46:45 on dist-test-slave-n1hw"
server_key: "f4b42364e11ea27cca1d0e479c60a1d8"
server_key_iv: "767749fab6d2f7751fca787f214b25ac"
server_key_version: "encryptionkey@0"
I20260926 16:46:45.833372 11594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:45.858072 11594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:45.861153 11594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:45.861372 11594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:45.868412 11594 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.124:44661
I20260926 16:46:45.868716 11661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.124:44661 every 8 connection(s)
I20260926 16:46:45.869453 11594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-2/data/info.pb
I20260926 16:46:45.870515 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 11594
I20260926 16:46:45.870613 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440924937351-21553-0/minicluster-data/master-2/wal/instance
I20260926 16:46:45.870919 21553 external_mini_cluster.cc:1428] Setting key de9e094ecb348856e037246db64a8bf2
I20260926 16:46:45.872611 11662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:45.876820 11662 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46369 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:46:45 dist-test-slave-n1hw krb5kdc[11428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.124@KRBTEST.COM
Sep 26 16:46:45 dist-test-slave-n1hw krb5kdc[11428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441205, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.124@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:46:45.885699 11665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:45.872363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51732 (local address 127.21.12.124:44661)
0926 16:46:45.872797 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:45.872813 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:45.872846 (+ 33us) server_negotiation.cc:407] Connection header received
0926 16:46:45.873474 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:45.873489 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:45.873863 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:45.874134 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:45.875050 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:45.876033 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:45.877190 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:45.877867 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:45.877983 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:45.880996 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:45.881001 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:45.881010 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:45.881034 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:45.882957 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:45.883391 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:45.883403 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:45.883407 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:45.883513 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:45.883795 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:45.883800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:45.883802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:45.884232 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:45.884417 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:45.884729 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 16:46:45.884985 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"spinlock_wait_cycles":1920,"thread_start_us":129,"threads_started":1}
I20260926 16:46:45.887708 11514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:45.878859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52712 (local address 127.21.12.126:46369)
0926 16:46:45.879080 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:45.879087 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:45.879273 (+ 186us) server_negotiation.cc:407] Connection header received
0926 16:46:45.879401 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:45.879405 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:45.879469 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:45.879562 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:45.880358 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:45.880884 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:45.881893 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:45.882103 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:45.882214 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:45.884328 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:45.884331 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:45.884334 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:45.884367 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:45.885965 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:45.886427 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:45.886433 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:45.886435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:45.886514 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:45.886794 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:45.886798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:45.886799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:45.886987 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:45.887093 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:45.887420 (+ 327us) server_negotiation.cc:299] Negotiation successful
0926 16:46:45.887538 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260926 16:46:45.887779 11510 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.124" port: 44661 } attempt: 3
I20260926 16:46:45.888584 11666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:45.878770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:46369 (local address 127.0.0.1:52712)
0926 16:46:45.879137 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:45.879184 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:45.879304 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:45.879614 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:45.879618 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:45.880047 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:46:45.880230 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:45.880238 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:45.881052 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:45.881055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:45.881689 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:45.881698 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:45.882213 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:45.882217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:45.882333 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:45.882839 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:46:45.882853 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:46:45.884191 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:46:45.886070 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:45.886079 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:45.886085 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:45.886304 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:45.886606 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:45.886611 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:45.886613 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:45.886695 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:45.887093 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:46:45.887100 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:46:45.887211 (+ 111us) client_negotiation.cc:769] Sending connection context
0926 16:46:45.887364 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 16:46:45.887576 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":121,"threads_started":1}
I20260926 16:46:45.890064 11662 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.125" port: 43535 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:46:45 dist-test-slave-n1hw krb5kdc[11428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441205, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for kudu/127.21.12.124@KRBTEST.COM
I20260926 16:46:45.893663 11586 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.124" port: 44661 } attempt: 3
Sep 26 16:46:45 dist-test-slave-n1hw krb5kdc[11428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
Sep 26 16:46:45 dist-test-slave-n1hw krb5kdc[11428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441205, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.124@KRBTEST.COM for kudu/127.21.12.125@KRBTEST.COM
I20260926 16:46:45.897723 11513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:45.888125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:44661 (local address 127.0.0.1:51744)
0926 16:46:45.888385 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:45.888397 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:45.888468 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:45.888930 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:45.888933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:45.889097 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:46:45.889184 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:45.889214 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:45.890073 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:45.890076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:45.890855 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:45.890863 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:45.891445 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:45.891449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:45.891548 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:45.892022 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:46:45.892036 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:46:45.893586 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:46:45.895776 (+ 2190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:45.895782 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:45.895787 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:45.896005 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:45.896360 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:45.896364 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:45.896367 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:45.896493 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:45.896976 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:46:45.896979 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:46:45.897094 (+ 115us) client_negotiation.cc:769] Sending connection context
0926 16:46:45.897209 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 16:46:45.897371 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215}
I20260926 16:46:45.898437 11665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:45.888175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51744 (local address 127.21.12.124:44661)
0926 16:46:45.888285 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:45.888293 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:45.888438 (+ 145us) server_negotiation.cc:407] Connection header received
0926 16:46:45.888561 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:45.888563 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:45.888792 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:45.888881 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:45.889317 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:45.889971 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:45.890988 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:45.891285 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:45.891453 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:45.893710 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:45.893715 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:45.893719 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:45.893741 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:45.895649 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:45.896100 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:45.896102 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:45.896103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:45.896208 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:45.896627 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:45.896630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:45.896632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:45.896860 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:45.896954 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:45.898159 (+ 1205us) server_negotiation.cc:299] Negotiation successful
0926 16:46:45.898275 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":39}
I20260926 16:46:45.899610 11514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:45.888270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52716 (local address 127.21.12.126:46369)
0926 16:46:45.888387 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:45.888393 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:45.888407 (+ 14us) server_negotiation.cc:407] Connection header received
0926 16:46:45.888442 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:45.888445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:45.888490 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:45.888564 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:45.889245 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:45.889970 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:45.891990 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:45.892352 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:45.892450 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:45.895322 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:45.895326 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:45.895328 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:45.895369 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:45.897852 (+ 2483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:45.898291 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:45.898296 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:45.898297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:45.898393 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:45.898687 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:45.898691 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:45.898692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:45.898882 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:45.899077 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:45.899222 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 16:46:45.899328 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42,"spinlock_wait_cycles":3328}
Sep 26 16:46:45 dist-test-slave-n1hw krb5kdc[11428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441205, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.125@KRBTEST.COM for kudu/127.21.12.124@KRBTEST.COM
I20260926 16:46:45.901964 11510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e6c4b13b244400684a8cd8a3b346d21: Bootstrap starting.
I20260926 16:46:45.902518 11592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:45.890638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47250 (local address 127.21.12.125:43535)
0926 16:46:45.890766 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:45.890769 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:45.890782 (+ 13us) server_negotiation.cc:407] Connection header received
0926 16:46:45.890820 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:45.890822 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:45.890856 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:45.890918 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:45.891379 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:45.892092 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:45.893000 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:45.893382 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:45.893472 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:45.895858 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:45.895863 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:45.895865 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:45.895889 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:45.897638 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:45.898256 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:45.898260 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:45.898262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:45.898352 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:45.898666 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:45.898670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:45.898672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:45.898876 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:45.898950 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:45.902267 (+ 3317us) server_negotiation.cc:299] Negotiation successful
0926 16:46:45.902382 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 16:46:45.902931 11666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:45.890541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.125:43535 (local address 127.0.0.1:47250)
0926 16:46:45.890688 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:45.890700 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:45.890759 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:45.890959 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:45.890962 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:45.891135 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:46:45.891246 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:45.891253 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:45.892202 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:45.892208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:45.892857 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:45.892867 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:45.893634 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:45.893640 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:45.893782 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:45.894234 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:46:45.894253 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:46:45.895747 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:46:45.897779 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:45.897785 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:45.897788 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:45.898031 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:45.898467 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:45.898473 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:45.898477 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:45.898573 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:45.901997 (+ 3424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:46:45.902000 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:46:45.902089 (+ 89us) client_negotiation.cc:769] Sending connection context
0926 16:46:45.902628 (+ 539us) client_negotiation.cc:240] Negotiation successful
0926 16:46:45.902796 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260926 16:46:45.904175 11662 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.124" port: 44661 } has no permanent_uuid. Determining permanent_uuid...
I20260926 16:46:45.905037 11510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e6c4b13b244400684a8cd8a3b346d21: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:45.906113 11510 log.cc:824] T 00000000000000000000000000000000 P 6e6c4b13b244400684a8cd8a3b346d21: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:45.906436 11667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:45.895291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51752 (local address 127.21.12.124:44661)
0926 16:46:45.896362 (+ 1071us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:45.896369 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:45.896384 (+ 15us) server_negotiation.cc:407] Connection header received
0926 16:46:45.896420 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:45.896425 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:45.896468 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:45.896577 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:45.897497 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:45.898336 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:45.899390 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:45.899755 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:45.900201 (+ 446us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:45.902566 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:45.902569 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:45.902572 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:45.902595 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:45.904612 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:45.905070 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:45.905073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:45.905075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:45.905155 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:45.905440 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:45.905443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:45.905445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:45.905628 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:45.906061 (+ 433us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:45.906137 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 16:46:45.906250 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":105,"threads_started":1}
I20260926 16:46:45.906505 11588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:45.894558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:44661 (local address 127.0.0.1:51752)
0926 16:46:45.895016 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:45.895031 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:45.895104 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:45.896612 (+ 1508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:45.896616 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:45.897251 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:46:45.897364 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:45.897372 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:45.898464 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:45.898467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:45.899253 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:45.899263 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:45.899881 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:45.899887 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:45.899984 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:45.900444 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:46:45.900465 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:46:45.901912 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:46:45.904716 (+ 2804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:45.904721 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:45.904723 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:45.904959 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:45.905250 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:45.905253 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:45.905255 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:45.905334 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:45.905744 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:46:45.905747 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:46:45.905860 (+ 113us) client_negotiation.cc:769] Sending connection context
0926 16:46:45.905963 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 16:46:45.906339 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":240}
I20260926 16:46:45.908994 11510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e6c4b13b244400684a8cd8a3b346d21: No bootstrap required, opened a new log
Sep 26 16:46:45 dist-test-slave-n1hw krb5kdc[11428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.125@KRBTEST.COM
I20260926 16:46:45.909647 11586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e77bfa21c6384f5e932d7f58218ae7ca: Bootstrap starting.
Sep 26 16:46:45 dist-test-slave-n1hw krb5kdc[11428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441205, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.124@KRBTEST.COM for kudu/127.21.12.124@KRBTEST.COM
I20260926 16:46:45.912670 11586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e77bfa21c6384f5e932d7f58218ae7ca: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:45.913434 11666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:45.904651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:44661 (local address 127.0.0.1:51766)
0926 16:46:45.905013 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:45.905025 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:45.905101 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:45.905448 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:45.905450 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:45.905597 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:46:45.905674 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:45.905680 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:45.906592 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:45.906595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:45.907251 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:45.907261 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:45.907875 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:45.907878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:45.907944 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:45.908344 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:46:45.908359 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:46:45.910084 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:46:45.911950 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:45.911952 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:45.911954 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:45.912174 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:45.912446 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:45.912448 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:45.912450 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:45.912536 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:45.912961 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:46:45.912964 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:46:45.913057 (+ 93us) client_negotiation.cc:769] Sending connection context
0926 16:46:45.913153 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 16:46:45.913268 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313}
I20260926 16:46:45.913655 11586 log.cc:824] T 00000000000000000000000000000000 P e77bfa21c6384f5e932d7f58218ae7ca: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:45.913985 11510 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e6c4b13b244400684a8cd8a3b346d21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e6c4b13b244400684a8cd8a3b346d21" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46369 } } peers { permanent_uuid: "e77bfa21c6384f5e932d7f58218ae7ca" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 43535 } } peers { permanent_uuid: "77a99ea0822f40c0ae7e9ec14affeb7d" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 44661 } }
I20260926 16:46:45.914012 11665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:45.904825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51766 (local address 127.21.12.124:44661)
0926 16:46:45.904945 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:45.904950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:45.905176 (+ 226us) server_negotiation.cc:407] Connection header received
0926 16:46:45.905200 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:45.905202 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:45.905281 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:45.905354 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:45.905808 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:45.906490 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:45.907372 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:45.907676 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:45.907778 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:45.910208 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:45.910212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:45.910214 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:45.910235 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:45.911803 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:45.912271 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:45.912273 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:45.912275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:45.912356 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:45.912635 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:45.912640 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:45.912641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:45.912837 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:45.912928 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:45.913776 (+ 848us) server_negotiation.cc:299] Negotiation successful
0926 16:46:45.913882 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 16:46:45.914443 11510 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e6c4b13b244400684a8cd8a3b346d21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e6c4b13b244400684a8cd8a3b346d21, State: Initialized, Role: FOLLOWER
I20260926 16:46:45.914534 11590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:45.901223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47254 (local address 127.21.12.125:43535)
0926 16:46:45.901341 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:45.901345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:45.901416 (+ 71us) server_negotiation.cc:407] Connection header received
0926 16:46:45.901584 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:45.901589 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:45.901630 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:45.901676 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:45.902513 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:45.904033 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:45.905700 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:45.906101 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:45.906180 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:45.910299 (+ 4119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:45.910303 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:45.910305 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:45.910328 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:45.912407 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:45.912884 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:45.912887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:45.912889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:45.912972 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:45.913253 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:45.913256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:45.913258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:45.913468 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:45.913627 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:45.914247 (+ 620us) server_negotiation.cc:299] Negotiation successful
0926 16:46:45.914361 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":49}
I20260926 16:46:45.915247 11510 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e6c4b13b244400684a8cd8a3b346d21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e6c4b13b244400684a8cd8a3b346d21" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46369 } } peers { permanent_uuid: "e77bfa21c6384f5e932d7f58218ae7ca" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 43535 } } peers { permanent_uuid: "77a99ea0822f40c0ae7e9ec14affeb7d" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 44661 } }
I20260926 16:46:45.915889 11586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e77bfa21c6384f5e932d7f58218ae7ca: No bootstrap required, opened a new log
I20260926 16:46:45.916707 11510 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e6c4b13b244400684a8cd8a3b346d21 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:46:45.916877 11662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77a99ea0822f40c0ae7e9ec14affeb7d: Bootstrap starting.
W20260926 16:46:45.917878 11463 tablet.cc:2369] T 00000000000000000000000000000000 P 6e6c4b13b244400684a8cd8a3b346d21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:46:45.919324 11662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77a99ea0822f40c0ae7e9ec14affeb7d: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:45.919651 21553 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:46:45.920208 11662 log.cc:824] T 00000000000000000000000000000000 P 77a99ea0822f40c0ae7e9ec14affeb7d: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:45.920935 11484 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "6e6c4b13b244400684a8cd8a3b346d21"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52716
I20260926 16:46:45.921149 11484 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e6c4b13b244400684a8cd8a3b346d21 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 16:46:45.921257 11484 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e6c4b13b244400684a8cd8a3b346d21 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:45.921490 11586 raft_consensus.cc:348] T 00000000000000000000000000000000 P e77bfa21c6384f5e932d7f58218ae7ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e6c4b13b244400684a8cd8a3b346d21" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46369 } } peers { permanent_uuid: "e77bfa21c6384f5e932d7f58218ae7ca" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 43535 } } peers { permanent_uuid: "77a99ea0822f40c0ae7e9ec14affeb7d" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 44661 } }
I20260926 16:46:45.921700 11586 raft_consensus.cc:729] T 00000000000000000000000000000000 P e77bfa21c6384f5e932d7f58218ae7ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e77bfa21c6384f5e932d7f58218ae7ca, State: Initialized, Role: FOLLOWER
I20260926 16:46:45.922042 11586 consensus_queue.cc:260] T 00000000000000000000000000000000 P e77bfa21c6384f5e932d7f58218ae7ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e6c4b13b244400684a8cd8a3b346d21" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46369 } } peers { permanent_uuid: "e77bfa21c6384f5e932d7f58218ae7ca" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 43535 } } peers { permanent_uuid: "77a99ea0822f40c0ae7e9ec14affeb7d" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 44661 } }
I20260926 16:46:45.922475 11683 sys_catalog.cc:455] T 00000000000000000000000000000000 P e77bfa21c6384f5e932d7f58218ae7ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e6c4b13b244400684a8cd8a3b346d21" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46369 } } peers { permanent_uuid: "e77bfa21c6384f5e932d7f58218ae7ca" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 43535 } } peers { permanent_uuid: "77a99ea0822f40c0ae7e9ec14affeb7d" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 44661 } } }
I20260926 16:46:45.922605 11683 sys_catalog.cc:458] T 00000000000000000000000000000000 P e77bfa21c6384f5e932d7f58218ae7ca [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:46:45.922575 11484 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e6c4b13b244400684a8cd8a3b346d21 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e6c4b13b244400684a8cd8a3b346d21" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46369 } } peers { permanent_uuid: "e77bfa21c6384f5e932d7f58218ae7ca" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 43535 } } peers { permanent_uuid: "77a99ea0822f40c0ae7e9ec14affeb7d" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 44661 } }
I20260926 16:46:45.923291 11586 sys_catalog.cc:564] T 00000000000000000000000000000000 P e77bfa21c6384f5e932d7f58218ae7ca [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:46:45.923426 11484 leader_election.cc:288] T 00000000000000000000000000000000 P 6e6c4b13b244400684a8cd8a3b346d21 [CANDIDATE]: Term 1 election: Requested vote from peers e77bfa21c6384f5e932d7f58218ae7ca (127.21.12.125:43535), 77a99ea0822f40c0ae7e9ec14affeb7d (127.21.12.124:44661)
I20260926 16:46:45.924300 11559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6e6c4b13b244400684a8cd8a3b346d21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e77bfa21c6384f5e932d7f58218ae7ca"
I20260926 16:46:45.924526 11559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e77bfa21c6384f5e932d7f58218ae7ca [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:45.925891 11559 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e77bfa21c6384f5e932d7f58218ae7ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e6c4b13b244400684a8cd8a3b346d21 in term 1.
I20260926 16:46:45.926522 11458 leader_election.cc:302] T 00000000000000000000000000000000 P 6e6c4b13b244400684a8cd8a3b346d21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e6c4b13b244400684a8cd8a3b346d21, e77bfa21c6384f5e932d7f58218ae7ca; no voters:
I20260926 16:46:45.928299 11662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77a99ea0822f40c0ae7e9ec14affeb7d: No bootstrap required, opened a new log
I20260926 16:46:45.932526 11662 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77a99ea0822f40c0ae7e9ec14affeb7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e6c4b13b244400684a8cd8a3b346d21" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46369 } } peers { permanent_uuid: "e77bfa21c6384f5e932d7f58218ae7ca" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 43535 } } peers { permanent_uuid: "77a99ea0822f40c0ae7e9ec14affeb7d" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 44661 } }
I20260926 16:46:45.926877 11442 master_runner.cc:186] Error getting master registration for 127.21.12.125:43535: OK, instance_id { permanent_uuid: "e77bfa21c6384f5e932d7f58218ae7ca" instance_seqno: 1790441205641095 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 16:46:45.934352 11442 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 16:46:45.934512 11670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e6c4b13b244400684a8cd8a3b346d21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e6c4b13b244400684a8cd8a3b346d21" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46369 } } peers { permanent_uuid: "e77bfa21c6384f5e932d7f58218ae7ca" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 43535 } } peers { permanent_uuid: "77a99ea0822f40c0ae7e9ec14affeb7d" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 44661 } } }
I20260926 16:46:45.934634 11670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e6c4b13b244400684a8cd8a3b346d21 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:46:45.934731 11670 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e6c4b13b244400684a8cd8a3b346d21 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:46:45.934816 11670 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e6c4b13b244400684a8cd8a3b346d21 [term 1 LEADER]: Becoming Leader. State: Replica: 6e6c4b13b244400684a8cd8a3b346d21, State: Running, Role: LEADER
I20260926 16:46:45.935145 11670 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e6c4b13b244400684a8cd8a3b346d21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e6c4b13b244400684a8cd8a3b346d21" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46369 } } peers { permanent_uuid: "e77bfa21c6384f5e932d7f58218ae7ca" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 43535 } } peers { permanent_uuid: "77a99ea0822f40c0ae7e9ec14affeb7d" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 44661 } }
W20260926 16:46:45.939030 11681 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6e6c4b13b244400684a8cd8a3b346d21: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260926 16:46:45.936839 11517 master_runner.cc:186] Error getting master registration for 127.21.12.124:44661: OK, instance_id { permanent_uuid: "77a99ea0822f40c0ae7e9ec14affeb7d" instance_seqno: 1790441205866463 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 16:46:45.940902 11517 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 16:46:45.941712 11699 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e77bfa21c6384f5e932d7f58218ae7ca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 16:46:45.941874 11699 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 16:46:45.940721 11700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e6c4b13b244400684a8cd8a3b346d21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e6c4b13b244400684a8cd8a3b346d21. Latest consensus state: current_term: 1 leader_uuid: "6e6c4b13b244400684a8cd8a3b346d21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e6c4b13b244400684a8cd8a3b346d21" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46369 } } peers { permanent_uuid: "e77bfa21c6384f5e932d7f58218ae7ca" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 43535 } } peers { permanent_uuid: "77a99ea0822f40c0ae7e9ec14affeb7d" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 44661 } } }
W20260926 16:46:45.942014 11681 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 16:46:45.942047 11700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e6c4b13b244400684a8cd8a3b346d21 [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:45.942538 11706 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:46:45.942708 11590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:45.927800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47264 (local address 127.21.12.125:43535)
0926 16:46:45.927901 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:45.927906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:45.930987 (+ 3081us) server_negotiation.cc:407] Connection header received
0926 16:46:45.931134 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:45.931138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:45.931181 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:45.931228 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:45.932823 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:45.933555 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:45.935435 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:45.935778 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:45.935856 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:45.939088 (+ 3232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:45.939092 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:45.939097 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:45.939123 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:45.940824 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:45.941322 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:45.941326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:45.941328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:45.941411 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:45.941735 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:45.941738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:45.941741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:45.941972 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:45.942090 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:45.942443 (+ 353us) server_negotiation.cc:299] Negotiation successful
0926 16:46:45.942561 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 16:46:45.937130 11665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:45.925588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51776 (local address 127.21.12.124:44661)
0926 16:46:45.925695 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:45.925700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:45.926196 (+ 496us) server_negotiation.cc:407] Connection header received
0926 16:46:45.926484 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:45.926489 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:45.926547 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:45.926606 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:45.928169 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:45.928687 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:45.929981 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:45.930317 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:45.930391 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:45.932254 (+ 1863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:45.932257 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:45.932260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:45.932283 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:45.934958 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:45.935387 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:45.935394 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:45.935399 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:45.935483 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:45.936178 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:45.936182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:45.936184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:45.936388 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:45.936501 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:45.936860 (+ 359us) server_negotiation.cc:299] Negotiation successful
0926 16:46:45.936979 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 16:46:45.943727 11662 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77a99ea0822f40c0ae7e9ec14affeb7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77a99ea0822f40c0ae7e9ec14affeb7d, State: Initialized, Role: FOLLOWER
I20260926 16:46:45.944036 11706 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:46:45.944074 11662 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77a99ea0822f40c0ae7e9ec14affeb7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e6c4b13b244400684a8cd8a3b346d21" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46369 } } peers { permanent_uuid: "e77bfa21c6384f5e932d7f58218ae7ca" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 43535 } } peers { permanent_uuid: "77a99ea0822f40c0ae7e9ec14affeb7d" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 44661 } }
I20260926 16:46:45.945397 11662 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77a99ea0822f40c0ae7e9ec14affeb7d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:46:45.945432 11636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6e6c4b13b244400684a8cd8a3b346d21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "77a99ea0822f40c0ae7e9ec14affeb7d"
I20260926 16:46:45.945677 11636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77a99ea0822f40c0ae7e9ec14affeb7d [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:45.947660 11636 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 77a99ea0822f40c0ae7e9ec14affeb7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e6c4b13b244400684a8cd8a3b346d21 in term 1.
I20260926 16:46:45.947956 11514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:45.926197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52730 (local address 127.21.12.126:46369)
0926 16:46:45.926353 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:45.926359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:45.930715 (+ 4356us) server_negotiation.cc:407] Connection header received
0926 16:46:45.930831 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:45.930834 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:45.930884 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:45.930966 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:45.932758 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:45.933552 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:45.942182 (+ 8630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:45.942518 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:45.942606 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:45.944315 (+ 1709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:45.944320 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:45.944323 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:45.944348 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:45.946071 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:45.946548 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:45.946554 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:45.946556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:45.946652 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:45.946986 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:45.946989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:45.946991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:45.947185 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:45.947286 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:45.947671 (+ 385us) server_negotiation.cc:299] Negotiation successful
0926 16:46:45.947791 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 16:46:45.948395 11707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77a99ea0822f40c0ae7e9ec14affeb7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e6c4b13b244400684a8cd8a3b346d21" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46369 } } peers { permanent_uuid: "e77bfa21c6384f5e932d7f58218ae7ca" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 43535 } } peers { permanent_uuid: "77a99ea0822f40c0ae7e9ec14affeb7d" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 44661 } } }
I20260926 16:46:45.948570 11707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77a99ea0822f40c0ae7e9ec14affeb7d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:46:45.950229 11636 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 77a99ea0822f40c0ae7e9ec14affeb7d [term 1 FOLLOWER]: Refusing update from remote peer 6e6c4b13b244400684a8cd8a3b346d21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:46:45.950385 11559 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e77bfa21c6384f5e932d7f58218ae7ca [term 1 FOLLOWER]: Refusing update from remote peer 6e6c4b13b244400684a8cd8a3b346d21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:46:45.950909 11670 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6e6c4b13b244400684a8cd8a3b346d21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77a99ea0822f40c0ae7e9ec14affeb7d" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 44661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:46:45.951278 11670 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6e6c4b13b244400684a8cd8a3b346d21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77bfa21c6384f5e932d7f58218ae7ca" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 43535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:46:45.954725 11707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77a99ea0822f40c0ae7e9ec14affeb7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e6c4b13b244400684a8cd8a3b346d21. Latest consensus state: current_term: 1 leader_uuid: "6e6c4b13b244400684a8cd8a3b346d21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e6c4b13b244400684a8cd8a3b346d21" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46369 } } peers { permanent_uuid: "e77bfa21c6384f5e932d7f58218ae7ca" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 43535 } } peers { permanent_uuid: "77a99ea0822f40c0ae7e9ec14affeb7d" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 44661 } } }
I20260926 16:46:45.954875 11707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77a99ea0822f40c0ae7e9ec14affeb7d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:46:45.955272 11683 sys_catalog.cc:455] T 00000000000000000000000000000000 P e77bfa21c6384f5e932d7f58218ae7ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e6c4b13b244400684a8cd8a3b346d21. Latest consensus state: current_term: 1 leader_uuid: "6e6c4b13b244400684a8cd8a3b346d21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e6c4b13b244400684a8cd8a3b346d21" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46369 } } peers { permanent_uuid: "e77bfa21c6384f5e932d7f58218ae7ca" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 43535 } } peers { permanent_uuid: "77a99ea0822f40c0ae7e9ec14affeb7d" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 44661 } } }
I20260926 16:46:45.955404 11683 sys_catalog.cc:458] T 00000000000000000000000000000000 P e77bfa21c6384f5e932d7f58218ae7ca [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:46:45.958539 11700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e6c4b13b244400684a8cd8a3b346d21 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6e6c4b13b244400684a8cd8a3b346d21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e6c4b13b244400684a8cd8a3b346d21" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46369 } } peers { permanent_uuid: "e77bfa21c6384f5e932d7f58218ae7ca" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 43535 } } peers { permanent_uuid: "77a99ea0822f40c0ae7e9ec14affeb7d" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 44661 } } }
I20260926 16:46:45.958640 11700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e6c4b13b244400684a8cd8a3b346d21 [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:45.958802 11708 mvcc.cc:205] Tried to move back new op lower bound from 7333647179566338048 to 7333647179519561728. Current Snapshot: MvccSnapshot[applied={T|T < 7333647179566338048}]
I20260926 16:46:45.958874 11683 sys_catalog.cc:455] T 00000000000000000000000000000000 P e77bfa21c6384f5e932d7f58218ae7ca [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6e6c4b13b244400684a8cd8a3b346d21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e6c4b13b244400684a8cd8a3b346d21" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46369 } } peers { permanent_uuid: "e77bfa21c6384f5e932d7f58218ae7ca" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 43535 } } peers { permanent_uuid: "77a99ea0822f40c0ae7e9ec14affeb7d" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 44661 } } }
I20260926 16:46:45.958971 11683 sys_catalog.cc:458] T 00000000000000000000000000000000 P e77bfa21c6384f5e932d7f58218ae7ca [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:46:45.961879 11710 mvcc.cc:205] Tried to move back new op lower bound from 7333647179566338048 to 7333647179519561728. Current Snapshot: MvccSnapshot[applied={T|T < 7333647179566338048}]
I20260926 16:46:45.963266 11706 catalog_manager.cc:1347] Generated new cluster ID: 50c89cc7a23f49f493359dc2425b7e6c
I20260926 16:46:45.963327 11706 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:46:45.968327 11670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e6c4b13b244400684a8cd8a3b346d21 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6e6c4b13b244400684a8cd8a3b346d21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e6c4b13b244400684a8cd8a3b346d21" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46369 } } peers { permanent_uuid: "e77bfa21c6384f5e932d7f58218ae7ca" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 43535 } } peers { permanent_uuid: "77a99ea0822f40c0ae7e9ec14affeb7d" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 44661 } } }
I20260926 16:46:45.968456 11670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e6c4b13b244400684a8cd8a3b346d21 [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:45.970090 11707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77a99ea0822f40c0ae7e9ec14affeb7d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6e6c4b13b244400684a8cd8a3b346d21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e6c4b13b244400684a8cd8a3b346d21" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 46369 } } peers { permanent_uuid: "e77bfa21c6384f5e932d7f58218ae7ca" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 43535 } } peers { permanent_uuid: "77a99ea0822f40c0ae7e9ec14affeb7d" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 44661 } } }
I20260926 16:46:45.970237 11707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77a99ea0822f40c0ae7e9ec14affeb7d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:46:45.970404 11709 mvcc.cc:205] Tried to move back new op lower bound from 7333647179566338048 to 7333647179519561728. Current Snapshot: MvccSnapshot[applied={T|T < 7333647179566338048}]
W20260926 16:46:45.970589 11723 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 77a99ea0822f40c0ae7e9ec14affeb7d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 16:46:45.970678 11723 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 16:46:45.992865 11706 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:46:45.994215 11706 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:46:46.001391 11706 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e6c4b13b244400684a8cd8a3b346d21: Generated new TSK 0
I20260926 16:46:46.002275 11706 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 16:46:46.026342 11665 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:46:46.023474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51780 (local address 127.21.12.124:44661)
0926 16:46:46.023590 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:46.023594 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:46.025269 (+ 1675us) server_negotiation.cc:407] Connection header received
0926 16:46:46.025468 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:46.025472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:46.025523 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:46.025574 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:46.026209 (+ 635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51780: BlockingRecv error: recv got EOF from 127.0.0.1:51780 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260926 16:46:46.027791 11514 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:46:46.024148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52740 (local address 127.21.12.126:46369)
0926 16:46:46.024256 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:46.024261 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:46.026955 (+ 2694us) server_negotiation.cc:407] Connection header received
0926 16:46:46.027119 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:46.027122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:46.027168 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:46.027216 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:46.027679 (+ 463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52740: BlockingRecv error: recv got EOF from 127.0.0.1:52740 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260926 16:46:46.029175 11590 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:46:46.024722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47272 (local address 127.21.12.125:43535)
0926 16:46:46.024827 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:46.024831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:46.028359 (+ 3528us) server_negotiation.cc:407] Connection header received
0926 16:46:46.028516 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:46.028519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:46.028564 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:46.028612 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:46.029070 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47272: BlockingRecv error: recv got EOF from 127.0.0.1:47272 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260926 16:46:46.035619 11590 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:46:46.034126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47274 (local address 127.21.12.125:43535)
0926 16:46:46.034227 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:46.034231 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:46.034716 (+ 485us) server_negotiation.cc:407] Connection header received
0926 16:46:46.034941 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:46.034944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:46.035014 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:46.035066 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:46.035526 (+ 460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47274: BlockingRecv error: recv got EOF from 127.0.0.1:47274 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260926 16:46:46.040611 11590 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:46:46.039292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47288 (local address 127.21.12.125:43535)
0926 16:46:46.039382 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:46.039385 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:46.039802 (+ 417us) server_negotiation.cc:407] Connection header received
0926 16:46:46.039989 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:46.039992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:46.040035 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:46.040083 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:46.040526 (+ 443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47288: BlockingRecv error: recv got EOF from 127.0.0.1:47288 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260926 16:46:46.046144 11590 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:46:46.044683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47300 (local address 127.21.12.125:43535)
0926 16:46:46.044769 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:46.044773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:46.045251 (+ 478us) server_negotiation.cc:407] Connection header received
0926 16:46:46.045422 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:46.045425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:46.045468 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:46.045513 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:46.046055 (+ 542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47300: BlockingRecv error: recv got EOF from 127.0.0.1:47300 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260926 16:46:46.053720 11590 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:46:46.052277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47308 (local address 127.21.12.125:43535)
0926 16:46:46.052407 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:46.052411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:46.052857 (+ 446us) server_negotiation.cc:407] Connection header received
0926 16:46:46.053034 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:46.053038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:46.053083 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:46.053131 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:46.053628 (+ 497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47308: BlockingRecv error: recv got EOF from 127.0.0.1:47308 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260926 16:46:46.065696 11590 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:46:46.064222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47324 (local address 127.21.12.125:43535)
0926 16:46:46.064328 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:46.064332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:46.064797 (+ 465us) server_negotiation.cc:407] Connection header received
0926 16:46:46.064991 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:46.064994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:46.065042 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:46.065091 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:46.065592 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47324: BlockingRecv error: recv got EOF from 127.0.0.1:47324 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260926 16:46:46.086385 11590 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:46:46.084575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47328 (local address 127.21.12.125:43535)
0926 16:46:46.084716 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:46.084720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:46.085336 (+ 616us) server_negotiation.cc:407] Connection header received
0926 16:46:46.085546 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:46.085550 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:46.085605 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:46.085661 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:46.086270 (+ 609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47328: BlockingRecv error: recv got EOF from 127.0.0.1:47328 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260926 16:46:46.122753 11590 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:46:46.121104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47340 (local address 127.21.12.125:43535)
0926 16:46:46.121223 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:46.121227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:46.121735 (+ 508us) server_negotiation.cc:407] Connection header received
0926 16:46:46.121996 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:46.122000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:46.122052 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:46.122102 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:46.122620 (+ 518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47340: BlockingRecv error: recv got EOF from 127.0.0.1:47340 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260926 16:46:46.190477 11590 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:46:46.189342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47350 (local address 127.21.12.125:43535)
0926 16:46:46.189524 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:46.189528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:46.189562 (+ 34us) server_negotiation.cc:407] Connection header received
0926 16:46:46.189715 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:46.189718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:46.189782 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:46.189861 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:46.190329 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47350: BlockingRecv error: recv got EOF from 127.0.0.1:47350 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260926 16:46:46.321671 11590 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:46:46.320463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47358 (local address 127.21.12.125:43535)
0926 16:46:46.320617 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:46.320622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:46.320751 (+ 129us) server_negotiation.cc:407] Connection header received
0926 16:46:46.320927 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:46.320930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:46.320991 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:46.321083 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:46.321553 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47358: BlockingRecv error: recv got EOF from 127.0.0.1:47358 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 16:46:46.581081 11590 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:46:46.579906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47370 (local address 127.21.12.125:43535)
0926 16:46:46.580054 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:46.580059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:46.580168 (+ 109us) server_negotiation.cc:407] Connection header received
0926 16:46:46.580345 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:46.580348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:46.580407 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:46.580508 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:46.580963 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47370: BlockingRecv error: recv got EOF from 127.0.0.1:47370 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 16:46:46.943295 11699 catalog_manager.cc:1259] Loaded cluster ID: 50c89cc7a23f49f493359dc2425b7e6c
I20260926 16:46:46.943380 11699 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e77bfa21c6384f5e932d7f58218ae7ca: loading cluster ID for follower catalog manager: success
I20260926 16:46:46.945068 11699 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e77bfa21c6384f5e932d7f58218ae7ca: acquiring CA information for follower catalog manager: success
I20260926 16:46:46.946400 11699 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e77bfa21c6384f5e932d7f58218ae7ca: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 16:46:46.972044 11723 catalog_manager.cc:1259] Loaded cluster ID: 50c89cc7a23f49f493359dc2425b7e6c
I20260926 16:46:46.972158 11723 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 77a99ea0822f40c0ae7e9ec14affeb7d: loading cluster ID for follower catalog manager: success
I20260926 16:46:46.973927 11723 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 77a99ea0822f40c0ae7e9ec14affeb7d: acquiring CA information for follower catalog manager: success
I20260926 16:46:46.975090 11723 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 77a99ea0822f40c0ae7e9ec14affeb7d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 16:46:47.099150 11795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:47.095340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47374 (local address 127.21.12.125:43535)
0926 16:46:47.095598 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:47.095602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:47.095619 (+ 17us) server_negotiation.cc:407] Connection header received
0926 16:46:47.095733 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:47.095736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:47.095855 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:47.095956 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:46:47.096244 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:47.096756 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:47.097591 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:47.097872 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:47.097951 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:47.098226 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:46:47.098657 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:46:47.098751 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:47.098870 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 16:46:47.098919 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":79,"threads_started":1}
I20260926 16:46:47.106021 11801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:47.102196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51784 (local address 127.21.12.124:44661)
0926 16:46:47.102443 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:47.102446 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:47.102460 (+ 14us) server_negotiation.cc:407] Connection header received
0926 16:46:47.102551 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:47.102553 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:47.102661 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:47.102756 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:46:47.102985 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:47.103586 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:47.104459 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:47.104714 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:47.104793 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:47.105061 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:46:47.105515 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:46:47.105665 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:47.105741 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 16:46:47.105819 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":83,"threads_started":1}
I20260926 16:46:47.113373 11795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:47.109837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47390 (local address 127.21.12.125:43535)
0926 16:46:47.109962 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:47.109966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:47.109984 (+ 18us) server_negotiation.cc:407] Connection header received
0926 16:46:47.110041 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:47.110046 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:47.110096 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:47.110169 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:46:47.110542 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:47.111338 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:47.112208 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:47.112475 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:47.112617 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:47.112864 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:46:47.112949 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:46:47.113020 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:47.113180 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 16:46:47.113242 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 16:46:47.113979 11801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:47.108982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51786 (local address 127.21.12.124:44661)
0926 16:46:47.109128 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:47.109132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:47.109882 (+ 750us) server_negotiation.cc:407] Connection header received
0926 16:46:47.110052 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:47.110055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:47.110102 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:47.110148 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:46:47.110466 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:47.111223 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:47.112697 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:47.112980 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:47.113110 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:47.113451 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:46:47.113528 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:46:47.113600 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:47.113776 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 16:46:47.113829 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 16:46:47.114912 11806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:47.108930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52748 (local address 127.21.12.126:46369)
0926 16:46:47.109190 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:47.109194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:47.109521 (+ 327us) server_negotiation.cc:407] Connection header received
0926 16:46:47.109721 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:47.109728 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:47.109931 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:47.110068 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:46:47.110246 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:47.111110 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:47.112770 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:47.113156 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:47.113264 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:47.113537 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:46:47.114103 (+ 566us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:46:47.114196 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:47.114682 (+ 486us) server_negotiation.cc:299] Negotiation successful
0926 16:46:47.114747 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":73,"threads_started":1}
I20260926 16:46:47.121187 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 11442
I20260926 16:46:47.127763 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 11517
I20260926 16:46:47.134203 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 11594
2026-09-26T16:46:47Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3050 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:46:47 dist-test-slave-n1hw krb5kdc[11812](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:46:47 dist-test-slave-n1hw krb5kdc[11812](info): set up 2 sockets
Sep 26 16:46:47 dist-test-slave-n1hw krb5kdc[11812](info): commencing operation
krb5kdc: starting...
W20260926 16:46:48.173224 21553 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/krb5kdc/test-user.keytab.
Sep 26 16:46:48 dist-test-slave-n1hw krb5kdc[11812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:46:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:46:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
Entry for principal kudu/127.21.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab.
I20260926 16:46:48.294487 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:41421
--webserver_interface=127.21.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:45883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:41421,127.21.12.125:42689,127.21.12.124:43503
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:46:48.405802 11825 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 16:46:48.406071 11825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:48.406095 11825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:48.406132 11825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:48.410015 11825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:46:48.410064 11825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:48.410084 11825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:48.410100 11825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:46:48.410115 11825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:46:48.415140 11825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:45883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:41421,127.21.12.125:42689,127.21.12.124:43503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.126.keytab
--principal=kudu/127.21.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.126:41421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:46:48.416092 11825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:48.417032 11825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:48.425406 11831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:48.425418 11833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:48.425457 11830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:48.425676 11825 server_base.cc:1034] running on GCE node
I20260926 16:46:48.426301 11825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:48.427371 11825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:48.428562 11825 hybrid_clock.cc:648] HybridClock initialized: now 1790441208428542 us; error 29 us; skew 500 ppm
Sep 26 16:46:48 dist-test-slave-n1hw krb5kdc[11812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441208, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:46:48.431326 11825 init.cc:377] Logged in from keytab as kudu/127.21.12.126@KRBTEST.COM (short username kudu)
I20260926 16:46:48.432170 11825 webserver.cc:460] Webserver started at http://127.21.12.126:37723/ using document root <none> and password file <none>
I20260926 16:46:48.432698 11825 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:48.432753 11825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:48.432909 11825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:46:48.434901 11825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-0/data/instance:
uuid: "75670c2011e1454eb6fa99e5110a4ced"
format_stamp: "Formatted at 2026-09-26 16:46:48 on dist-test-slave-n1hw"
server_key: "47f4f0a2ee7b31e994eeead23e70cb7e"
server_key_iv: "e4768368b9849ef7453616ad3f3a644e"
server_key_version: "encryptionkey@0"
I20260926 16:46:48.435417 11825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-0/wal/instance:
uuid: "75670c2011e1454eb6fa99e5110a4ced"
format_stamp: "Formatted at 2026-09-26 16:46:48 on dist-test-slave-n1hw"
server_key: "47f4f0a2ee7b31e994eeead23e70cb7e"
server_key_iv: "e4768368b9849ef7453616ad3f3a644e"
server_key_version: "encryptionkey@0"
I20260926 16:46:48.438946 11825 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:46:48.441372 11840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:48.442648 11825 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:46:48.442771 11825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-0/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-0/wal
uuid: "75670c2011e1454eb6fa99e5110a4ced"
format_stamp: "Formatted at 2026-09-26 16:46:48 on dist-test-slave-n1hw"
server_key: "47f4f0a2ee7b31e994eeead23e70cb7e"
server_key_iv: "e4768368b9849ef7453616ad3f3a644e"
server_key_version: "encryptionkey@0"
I20260926 16:46:48.442874 11825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:48.455399 11825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:48.458200 11825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:48.458400 11825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:48.465315 11825 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.126:41421
I20260926 16:46:48.465634 11892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.126:41421 every 8 connection(s)
I20260926 16:46:48.466363 11825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-0/data/info.pb
I20260926 16:46:48.469370 11893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:48.470585 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 11825
I20260926 16:46:48.470675 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-0/wal/instance
I20260926 16:46:48.471027 21553 external_mini_cluster.cc:1428] Setting key 6ddeda88c4511bc3bec4c0f8145ae154
I20260926 16:46:48.474110 11893 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.126" port: 41421 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.12.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.125@KRBTEST.COM" created.
Sep 26 16:46:48 dist-test-slave-n1hw krb5kdc[11812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441208, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:46:48.490738 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:48.477795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43296 (local address 127.21.12.126:41421)
0926 16:46:48.478097 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:48.478109 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:48.478153 (+ 44us) server_negotiation.cc:407] Connection header received
0926 16:46:48.478753 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:48.478759 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:48.478991 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:48.479202 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:48.480122 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.481130 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.482402 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.482885 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.483076 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:48.485508 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:48.485523 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:48.485533 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:48.485554 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:48.487424 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:48.487945 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:48.487950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:48.487952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:48.488065 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:48.488402 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:48.488405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:48.488407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:48.488802 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:48.488957 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:48.489351 (+ 394us) server_negotiation.cc:299] Negotiation successful
0926 16:46:48.489615 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":91,"threads_started":1}
I20260926 16:46:48.490738 11896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:48.476794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:41421 (local address 127.0.0.1:43296)
0926 16:46:48.477345 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:48.477390 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:48.478437 (+ 1047us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:48.479197 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:48.479202 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:48.479671 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:46:48.479974 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:48.479989 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.481264 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.481270 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:48.482254 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:48.482264 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.483059 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.483062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:48.483159 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:48.483826 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:46:48.483842 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:46:48.485357 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:46:48.487558 (+ 2201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:48.487569 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:48.487579 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:48.487820 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:48.488172 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:48.488175 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:48.488177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:48.488295 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:48.488926 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:46:48.488933 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:46:48.489086 (+ 153us) client_negotiation.cc:769] Sending connection context
0926 16:46:48.489270 (+ 184us) client_negotiation.cc:240] Negotiation successful
0926 16:46:48.489668 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":113,"thread_start_us":153,"threads_started":1}
I20260926 16:46:48.493448 11893 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.125" port: 42689 } has no permanent_uuid. Determining permanent_uuid...
I20260926 16:46:48.494177 11896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:48.493917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.125:42689 (local address 127.0.0.1:49068)
0926 16:46:48.494038 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:48.494117 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.125:42689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 16:46:48.494406 11841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.12.125:42689: connect: Connection refused (error 111)
I20260926 16:46:48.495296 11896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:48.495040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.125:42689 (local address 127.0.0.1:49082)
0926 16:46:48.495196 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:48.495242 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.125:42689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260926 16:46:48.495466 11893 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.125:42689: Network error: Client connection negotiation failed: client connection to 127.21.12.125:42689: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.125.keytab.
Entry for principal kudu/127.21.12.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.125.keytab.
I20260926 16:46:48.497159 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.125:42689
--webserver_interface=127.21.12.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.125.keytab
--principal=kudu/127.21.12.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:45883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:41421,127.21.12.125:42689,127.21.12.124:43503
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:46:48.544777 11893 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.125" port: 42689 } attempt: 1
I20260926 16:46:48.545779 11896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:48.545422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.125:42689 (local address 127.0.0.1:49098)
0926 16:46:48.545596 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:48.545664 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.125:42689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 16:46:48.546571 11896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:48.546329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.125:42689 (local address 127.0.0.1:49104)
0926 16:46:48.546458 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:48.546516 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.125:42689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260926 16:46:48.546705 11893 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.125:42689: Network error: Client connection negotiation failed: client connection to 127.21.12.125:42689: connect: Connection refused (error 111)
W20260926 16:46:48.606091 11900 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 16:46:48.606407 11900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:48.606483 11900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:48.606562 11900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:48.610559 11900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:46:48.610634 11900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:48.610671 11900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:48.610726 11900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:46:48.610765 11900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:46:48.614879 11893 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.125" port: 42689 } attempt: 2
I20260926 16:46:48.615723 11896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:48.615415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.125:42689 (local address 127.0.0.1:49110)
0926 16:46:48.615571 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:48.615648 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.125:42689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 16:46:48.616026 11900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:45883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:41421,127.21.12.125:42689,127.21.12.124:43503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.125.keytab
--principal=kudu/127.21.12.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.125:42689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.12.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:46:48.616668 11896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:48.616427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.125:42689 (local address 127.0.0.1:49126)
0926 16:46:48.616552 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:48.616613 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.125:42689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 16:46:48.616823 11893 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.125:42689: Network error: Client connection negotiation failed: client connection to 127.21.12.125:42689: connect: Connection refused (error 111)
I20260926 16:46:48.617183 11900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:48.618212 11900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:48.626310 11906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:48.626395 11905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:48.626557 11900 server_base.cc:1034] running on GCE node
W20260926 16:46:48.626714 11908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:48.627314 11900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:48.628423 11900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:48.629658 11900 hybrid_clock.cc:648] HybridClock initialized: now 1790441208629632 us; error 23 us; skew 500 ppm
Sep 26 16:46:48 dist-test-slave-n1hw krb5kdc[11812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441208, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:46:48.632444 11900 init.cc:377] Logged in from keytab as kudu/127.21.12.125@KRBTEST.COM (short username kudu)
I20260926 16:46:48.633239 11900 webserver.cc:460] Webserver started at http://127.21.12.125:41763/ using document root <none> and password file <none>
I20260926 16:46:48.633724 11900 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:48.633852 11900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:48.634030 11900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:46:48.635938 11900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-1/data/instance:
uuid: "ba98563b5e644facb8f57d7cd32ec15e"
format_stamp: "Formatted at 2026-09-26 16:46:48 on dist-test-slave-n1hw"
server_key: "63a74522469dd8658491170369940bd0"
server_key_iv: "cfc100a643a1b2603dc6545dcc84facd"
server_key_version: "encryptionkey@0"
I20260926 16:46:48.636438 11900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-1/wal/instance:
uuid: "ba98563b5e644facb8f57d7cd32ec15e"
format_stamp: "Formatted at 2026-09-26 16:46:48 on dist-test-slave-n1hw"
server_key: "63a74522469dd8658491170369940bd0"
server_key_iv: "cfc100a643a1b2603dc6545dcc84facd"
server_key_version: "encryptionkey@0"
I20260926 16:46:48.639968 11900 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 16:46:48.642407 11915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:48.643736 11900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:46:48.643934 11900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-1/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-1/wal
uuid: "ba98563b5e644facb8f57d7cd32ec15e"
format_stamp: "Formatted at 2026-09-26 16:46:48 on dist-test-slave-n1hw"
server_key: "63a74522469dd8658491170369940bd0"
server_key_iv: "cfc100a643a1b2603dc6545dcc84facd"
server_key_version: "encryptionkey@0"
I20260926 16:46:48.644037 11900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:48.673426 11900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:48.676966 11900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:48.677187 11900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:48.684916 11900 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.125:42689
I20260926 16:46:48.685197 11967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.125:42689 every 8 connection(s)
I20260926 16:46:48.686007 11900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-1/data/info.pb
I20260926 16:46:48.688746 11968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:48.692507 11968 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.126" port: 41421 } has no permanent_uuid. Determining permanent_uuid...
I20260926 16:46:48.693461 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 11900
I20260926 16:46:48.693554 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-1/wal/instance
I20260926 16:46:48.693939 21553 external_mini_cluster.cc:1428] Setting key 498d6f086cb7f24faebb3d2943be21fa
WARNING: no policy specified for kudu/127.21.12.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.124@KRBTEST.COM" created.
Sep 26 16:46:48 dist-test-slave-n1hw krb5kdc[11812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441208, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.125@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:46:48.707993 11893 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.125" port: 42689 } attempt: 3
I20260926 16:46:48.710597 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:48.694485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43298 (local address 127.21.12.126:41421)
0926 16:46:48.694613 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:48.694617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:48.695595 (+ 978us) server_negotiation.cc:407] Connection header received
0926 16:46:48.697001 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:48.697005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:48.697066 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:48.697147 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:48.698424 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.698972 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.700087 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.700322 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.700436 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:48.705166 (+ 4730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:48.705191 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:48.705198 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:48.705236 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:48.707078 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:48.707644 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:48.707650 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:48.707652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:48.707738 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:48.708129 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:48.708134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:48.708136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:48.708436 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:48.710209 (+ 1773us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:48.710292 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 16:46:48.710430 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 16:46:48.710618 11970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:48.694953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:41421 (local address 127.0.0.1:43298)
0926 16:46:48.695430 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:48.695473 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:48.696749 (+ 1276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:48.697307 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:48.697314 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:48.697656 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:46:48.698279 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:48.698293 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.699106 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.699110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:48.699955 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:48.699965 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.700472 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.700476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:48.700642 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:48.701235 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:46:48.701251 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:46:48.704962 (+ 3711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:46:48.707209 (+ 2247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:48.707215 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:48.707232 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:48.707528 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:48.707847 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:48.707857 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:48.707859 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:48.708020 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:48.708574 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:46:48.708581 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:46:48.708937 (+ 356us) client_negotiation.cc:769] Sending connection context
0926 16:46:48.709142 (+ 205us) client_negotiation.cc:240] Negotiation successful
0926 16:46:48.709511 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":148,"threads_started":1}
I20260926 16:46:48.712212 11968 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.125" port: 42689 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:46:48 dist-test-slave-n1hw krb5kdc[11812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441208, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for kudu/127.21.12.125@KRBTEST.COM
Sep 26 16:46:48 dist-test-slave-n1hw krb5kdc[11812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441208, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.125@KRBTEST.COM for kudu/127.21.12.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.124.keytab.
Entry for principal kudu/127.21.12.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.124.keytab.
I20260926 16:46:48.720495 11896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:48.708519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.125:42689 (local address 127.0.0.1:49128)
0926 16:46:48.708726 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:48.708744 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:48.708829 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:48.709558 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:48.709564 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:48.709800 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:46:48.709935 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:48.709943 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.711763 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.711770 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:48.713112 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:48.713122 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.713816 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.713820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:48.713959 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:48.714496 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:46:48.714516 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:46:48.716046 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:46:48.718586 (+ 2540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:48.718591 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:48.718594 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:48.718863 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:48.719286 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:48.719289 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:48.719290 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:48.719412 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:48.719901 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:46:48.719906 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:46:48.720031 (+ 125us) client_negotiation.cc:769] Sending connection context
0926 16:46:48.720155 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 16:46:48.720300 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122}
I20260926 16:46:48.720721 11973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:48.708685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49128 (local address 127.21.12.125:42689)
0926 16:46:48.709096 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:48.709110 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:48.709132 (+ 22us) server_negotiation.cc:407] Connection header received
0926 16:46:48.709173 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:48.709179 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:48.709395 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:48.709552 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:48.710068 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.711586 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.713282 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.713658 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.713801 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:48.716214 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:48.716226 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:48.716243 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:48.716271 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:48.718414 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:48.718996 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:48.719010 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:48.719011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:48.719181 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:48.719536 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:48.719539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:48.719540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:48.719780 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:48.719894 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:48.720275 (+ 381us) server_negotiation.cc:299] Negotiation successful
0926 16:46:48.720463 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":127,"threads_started":1}
I20260926 16:46:48.721541 21553 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
/tmp/dist-test-taskEZzU7d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.124:43503
--webserver_interface=127.21.12.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.124.keytab
--principal=kudu/127.21.12.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.84:45883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.126:41421,127.21.12.125:42689,127.21.12.124:43503
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:46:48.722950 11893 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.124" port: 43503 } has no permanent_uuid. Determining permanent_uuid...
I20260926 16:46:48.723826 11896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:48.723503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:43503 (local address 127.0.0.1:33394)
0926 16:46:48.723656 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:48.723727 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.124:43503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 16:46:48.724641 11896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:48.724391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:43503 (local address 127.0.0.1:33402)
0926 16:46:48.724548 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:48.724591 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.124:43503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260926 16:46:48.724793 11893 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.124:43503: Network error: Client connection negotiation failed: client connection to 127.21.12.124:43503: connect: Connection refused (error 111)
I20260926 16:46:48.725577 11970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:48.712750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.125:42689 (local address 127.0.0.1:49142)
0926 16:46:48.712854 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:48.712868 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:48.712967 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:48.713373 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:48.713378 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:48.713556 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:46:48.713631 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:48.713636 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.714672 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.714678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:48.715526 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:48.715538 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.716150 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.716154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:48.716261 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:48.716682 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:46:48.716693 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:46:48.718414 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:46:48.723992 (+ 5578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:48.723998 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:48.724001 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:48.724284 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:48.724604 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:48.724610 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:48.724615 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:48.724714 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:48.725163 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:46:48.725165 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:46:48.725248 (+ 83us) client_negotiation.cc:769] Sending connection context
0926 16:46:48.725365 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 16:46:48.725470 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 16:46:48.725863 11974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:48.712801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49142 (local address 127.21.12.125:42689)
0926 16:46:48.713073 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:48.713081 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:48.713093 (+ 12us) server_negotiation.cc:407] Connection header received
0926 16:46:48.713129 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:48.713131 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:48.713163 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:48.713349 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:48.713747 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.714558 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.715649 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.716027 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.716112 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:48.718607 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:48.718610 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:48.718613 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:48.718638 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:48.723488 (+ 4850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:48.724393 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:48.724396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:48.724398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:48.724491 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:48.724810 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:48.724816 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:48.724820 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:48.725055 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:48.725157 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:48.725445 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 16:46:48.725579 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":92,"threads_started":1}
I20260926 16:46:48.726502 11968 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.124" port: 43503 } has no permanent_uuid. Determining permanent_uuid...
I20260926 16:46:48.727200 11970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:48.726963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:43503 (local address 127.0.0.1:33414)
0926 16:46:48.727079 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:48.727146 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.124:43503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 16:46:48.727447 11918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.12.124:43503: connect: Connection refused (error 111)
I20260926 16:46:48.728358 11970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:48.728137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:43503 (local address 127.0.0.1:33416)
0926 16:46:48.728245 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:48.728288 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.124:43503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 16:46:48.728586 11968 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.124:43503: Network error: Client connection negotiation failed: client connection to 127.21.12.124:43503: connect: Connection refused (error 111)
I20260926 16:46:48.756011 11893 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.124" port: 43503 } attempt: 1
I20260926 16:46:48.756965 11896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:48.756612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:43503 (local address 127.0.0.1:33420)
0926 16:46:48.756774 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:48.756871 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.124:43503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 16:46:48.757735 11896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:48.757500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:43503 (local address 127.0.0.1:33422)
0926 16:46:48.757621 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:48.757677 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.124:43503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 16:46:48.757930 11893 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.124:43503: Network error: Client connection negotiation failed: client connection to 127.21.12.124:43503: connect: Connection refused (error 111)
I20260926 16:46:48.777860 11968 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.124" port: 43503 } attempt: 1
I20260926 16:46:48.778849 11970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:48.778458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:43503 (local address 127.0.0.1:33428)
0926 16:46:48.778637 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:48.778730 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.124:43503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 16:46:48.779662 11970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:48.779419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:43503 (local address 127.0.0.1:33430)
0926 16:46:48.779538 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:48.779596 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.124:43503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 16:46:48.779855 11968 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.124:43503: Network error: Client connection negotiation failed: client connection to 127.21.12.124:43503: connect: Connection refused (error 111)
I20260926 16:46:48.833105 11893 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.124" port: 43503 } attempt: 2
I20260926 16:46:48.833969 11896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:48.833638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:43503 (local address 127.0.0.1:33438)
0926 16:46:48.833829 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:48.833895 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.124:43503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 16:46:48.834708 11896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:48.834505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:43503 (local address 127.0.0.1:33450)
0926 16:46:48.834609 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:48.834658 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.124:43503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 16:46:48.834887 11893 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.124:43503: Network error: Client connection negotiation failed: client connection to 127.21.12.124:43503: connect: Connection refused (error 111)
W20260926 16:46:48.835258 11975 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 16:46:48.835500 11975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:48.835556 11975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:48.835644 11975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:48.839710 11975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:46:48.839762 11975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:48.839781 11975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:48.839797 11975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:46:48.839838 11975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:46:48.845031 11975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.84:45883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.126:41421,127.21.12.125:42689,127.21.12.124:43503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/krb5kdc/kudu_127.21.12.124.keytab
--principal=kudu/127.21.12.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.124:43503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.12.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 15:58:58 UTC on ccycloud.quasar-ylhwwb.root.comops.site
build id 4674727
I20260926 16:46:48.846159 11975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:48.847194 11975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 16:46:48.848026 11968 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.124" port: 43503 } attempt: 2
I20260926 16:46:48.848871 11970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:48.848544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:43503 (local address 127.0.0.1:33466)
0926 16:46:48.848682 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:48.848753 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.124:43503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 16:46:48.849598 11970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:48.849359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:43503 (local address 127.0.0.1:33478)
0926 16:46:48.849474 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:48.849537 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.124:43503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 16:46:48.849735 11968 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.124:43503: Network error: Client connection negotiation failed: client connection to 127.21.12.124:43503: connect: Connection refused (error 111)
W20260926 16:46:48.856325 11981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:48.856318 11984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:48.856318 11982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:48.856594 11975 server_base.cc:1034] running on GCE node
I20260926 16:46:48.857254 11975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:48.858501 11975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:48.859714 11975 hybrid_clock.cc:648] HybridClock initialized: now 1790441208859695 us; error 29 us; skew 500 ppm
Sep 26 16:46:48 dist-test-slave-n1hw krb5kdc[11812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441208, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:46:48.862498 11975 init.cc:377] Logged in from keytab as kudu/127.21.12.124@KRBTEST.COM (short username kudu)
I20260926 16:46:48.863433 11975 webserver.cc:460] Webserver started at http://127.21.12.124:38187/ using document root <none> and password file <none>
I20260926 16:46:48.863996 11975 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:48.864108 11975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:48.864344 11975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:46:48.866423 11975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-2/data/instance:
uuid: "24b312a595fe4d7e9a4556550e07fcd7"
format_stamp: "Formatted at 2026-09-26 16:46:48 on dist-test-slave-n1hw"
server_key: "19f906f3456346285df1b6bf6c78ec3b"
server_key_iv: "927ab3b3c97cc116071ba3e4beb73e8d"
server_key_version: "encryptionkey@0"
I20260926 16:46:48.866961 11975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-2/wal/instance:
uuid: "24b312a595fe4d7e9a4556550e07fcd7"
format_stamp: "Formatted at 2026-09-26 16:46:48 on dist-test-slave-n1hw"
server_key: "19f906f3456346285df1b6bf6c78ec3b"
server_key_iv: "927ab3b3c97cc116071ba3e4beb73e8d"
server_key_version: "encryptionkey@0"
I20260926 16:46:48.870764 11975 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 16:46:48.873128 11991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:48.874399 11975 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 16:46:48.874545 11975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-2/data,/tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-2/wal
uuid: "24b312a595fe4d7e9a4556550e07fcd7"
format_stamp: "Formatted at 2026-09-26 16:46:48 on dist-test-slave-n1hw"
server_key: "19f906f3456346285df1b6bf6c78ec3b"
server_key_iv: "927ab3b3c97cc116071ba3e4beb73e8d"
server_key_version: "encryptionkey@0"
I20260926 16:46:48.874655 11975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:48.892448 11975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:48.895586 11975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:48.895792 11975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:48.902784 11975 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.124:43503
I20260926 16:46:48.903079 12043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.124:43503 every 8 connection(s)
I20260926 16:46:48.903749 11975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-2/data/info.pb
I20260926 16:46:48.906453 12044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:48.909140 21553 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu as pid 11975
I20260926 16:46:48.909236 21553 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEZzU7d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440924937351-21553-0/minicluster-data/master-2/wal/instance
I20260926 16:46:48.909561 21553 external_mini_cluster.cc:1428] Setting key 33d32cd96f496c0277db9c954652c611
I20260926 16:46:48.910184 12044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.126" port: 41421 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:46:48 dist-test-slave-n1hw krb5kdc[11812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441208, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.124@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
Sep 26 16:46:48 dist-test-slave-n1hw krb5kdc[11812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.124@KRBTEST.COM
I20260926 16:46:48.922221 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:48.912071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43308 (local address 127.21.12.126:41421)
0926 16:46:48.912200 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:48.912205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:48.912508 (+ 303us) server_negotiation.cc:407] Connection header received
0926 16:46:48.912836 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:48.912839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:48.912899 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:48.912983 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:48.913730 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.914288 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.915264 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.915479 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.915586 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:48.918220 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:48.918224 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:48.918227 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:48.918255 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:48.919941 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:48.920551 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:48.920555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:48.920557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:48.920641 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:48.920999 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:48.921002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:48.921003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:48.921257 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:48.921356 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:48.921922 (+ 566us) server_negotiation.cc:299] Negotiation successful
0926 16:46:48.922049 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 16:46:48.923141 12048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:48.912012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.126:41421 (local address 127.0.0.1:43308)
0926 16:46:48.912379 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:48.912416 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:48.912653 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:48.913033 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:48.913038 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:48.913362 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:46:48.913594 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:48.913605 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.914412 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.914417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:48.915136 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:48.915144 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.915617 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.915621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:48.915779 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:48.916356 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:46:48.916379 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:46:48.918053 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:46:48.920067 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:48.920075 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:48.920086 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:48.920455 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:48.920745 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:48.920747 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:48.920748 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:48.920898 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:48.921360 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:46:48.921367 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:46:48.921654 (+ 287us) client_negotiation.cc:769] Sending connection context
0926 16:46:48.921840 (+ 186us) client_negotiation.cc:240] Negotiation successful
0926 16:46:48.922150 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":120,"threads_started":1}
I20260926 16:46:48.923141 12047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:48.910945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33494 (local address 127.21.12.124:43503)
0926 16:46:48.911353 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:48.911366 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:48.911401 (+ 35us) server_negotiation.cc:407] Connection header received
0926 16:46:48.912296 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:48.912318 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:48.912668 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:48.912887 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:48.913709 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.914594 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.915408 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.915760 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.915875 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:48.918702 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:48.918707 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:48.918716 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:48.918736 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:48.920434 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:48.920936 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:48.920944 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:48.920946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:48.921031 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:48.921295 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:48.921297 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:48.921298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:48.921666 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:48.921886 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:48.922179 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 16:46:48.922295 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":279,"thread_start_us":126,"threads_started":1}
I20260926 16:46:48.924446 12044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.125" port: 42689 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:46:48 dist-test-slave-n1hw krb5kdc[11812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441208, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.124@KRBTEST.COM for kudu/127.21.12.125@KRBTEST.COM
Sep 26 16:46:48 dist-test-slave-n1hw krb5kdc[11812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.126@KRBTEST.COM
I20260926 16:46:48.933089 12048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:48.924904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.125:42689 (local address 127.0.0.1:49146)
0926 16:46:48.925072 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:48.925085 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:48.925150 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:48.925409 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:48.925415 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:48.925586 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:46:48.925705 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:48.925712 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.926530 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.926536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:48.927074 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:48.927082 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.927603 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.927606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:48.927701 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:48.928229 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:46:48.928243 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:46:48.929453 (+ 1210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:46:48.931575 (+ 2122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:48.931577 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:48.931579 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:48.931764 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:48.932093 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:48.932095 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:48.932096 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:48.932167 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:48.932583 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:46:48.932586 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:46:48.932683 (+ 97us) client_negotiation.cc:769] Sending connection context
0926 16:46:48.932799 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 16:46:48.932933 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260926 16:46:48.933135 11974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:48.925055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49146 (local address 127.21.12.125:42689)
0926 16:46:48.925176 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:48.925180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:48.925193 (+ 13us) server_negotiation.cc:407] Connection header received
0926 16:46:48.925236 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:48.925239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:48.925288 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:48.925382 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:48.925856 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.926415 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.927198 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.927471 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.927605 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:48.929587 (+ 1982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:48.929590 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:48.929592 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:48.929614 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:48.931428 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:48.931866 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:48.931869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:48.931871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:48.931961 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:48.932257 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:48.932263 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:48.932267 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:48.932467 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:48.932566 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:48.932860 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 16:46:48.932989 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 16:46:48.933970 12044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.124" port: 43503 } has no permanent_uuid. Determining permanent_uuid...
I20260926 16:46:48.934070 11893 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.124" port: 43503 } attempt: 3
I20260926 16:46:48.934368 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:48.925601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43320 (local address 127.21.12.126:41421)
0926 16:46:48.925694 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:48.925698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:48.925784 (+ 86us) server_negotiation.cc:407] Connection header received
0926 16:46:48.925925 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:48.925927 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:48.925968 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:48.926055 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:48.926615 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.927107 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.927857 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.928075 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.928181 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:48.930614 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:48.930617 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:48.930620 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:48.930665 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:48.932228 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:48.932601 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:48.932604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:48.932606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:48.932684 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:48.932974 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:48.932977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:48.932979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:48.933167 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:48.933456 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:48.933818 (+ 362us) server_negotiation.cc:299] Negotiation successful
0926 16:46:48.933933 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Sep 26 16:46:48 dist-test-slave-n1hw krb5kdc[11812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441208, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.126@KRBTEST.COM for kudu/127.21.12.124@KRBTEST.COM
Sep 26 16:46:48 dist-test-slave-n1hw krb5kdc[11812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.125@KRBTEST.COM
I20260926 16:46:48.940918 11968 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.124" port: 43503 } attempt: 3
Sep 26 16:46:48 dist-test-slave-n1hw krb5kdc[11812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441208, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.124@KRBTEST.COM for kudu/127.21.12.124@KRBTEST.COM
I20260926 16:46:48.944600 11896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:48.934437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:43503 (local address 127.0.0.1:33510)
0926 16:46:48.934557 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:48.934571 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:48.934645 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:48.935126 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:48.935129 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:48.935300 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:46:48.935426 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:48.935434 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.936487 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.936492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:48.937366 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:48.937377 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.938041 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.938045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:48.938138 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:48.938581 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:46:48.938593 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:46:48.940083 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:46:48.943007 (+ 2924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:48.943010 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:48.943012 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:48.943307 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:48.943608 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:48.943612 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:48.943614 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:48.943702 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:48.944131 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:46:48.944135 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:46:48.944235 (+ 100us) client_negotiation.cc:769] Sending connection context
0926 16:46:48.944318 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 16:46:48.944447 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 16:46:48.945503 12047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:48.934810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33510 (local address 127.21.12.124:43503)
0926 16:46:48.934919 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:48.934923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:48.934936 (+ 13us) server_negotiation.cc:407] Connection header received
0926 16:46:48.934968 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:48.934971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:48.935011 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:48.935110 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:48.935522 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.936368 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.937504 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.937903 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.938327 (+ 424us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:48.940217 (+ 1890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:48.940222 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:48.940226 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:48.940250 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:48.942882 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:48.943422 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:48.943425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:48.943426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:48.943505 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:48.943780 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:48.943785 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:48.943787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:48.943994 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:48.945092 (+ 1098us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:48.945175 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 16:46:48.945355 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 16:46:48.945514 11974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:48.935519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49158 (local address 127.21.12.125:42689)
0926 16:46:48.935614 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:48.935620 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:48.935631 (+ 11us) server_negotiation.cc:407] Connection header received
0926 16:46:48.935764 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:48.935767 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:48.935813 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:48.935892 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:48.936459 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.936921 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.937653 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.938010 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.938156 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:48.941051 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:48.941054 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:48.941057 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:48.941081 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:48.943246 (+ 2165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:48.943762 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:48.943766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:48.943769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:48.943860 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:48.944173 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:48.944176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:48.944177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:48.944377 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:48.944592 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:48.945261 (+ 669us) server_negotiation.cc:299] Negotiation successful
0926 16:46:48.945377 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":31}
I20260926 16:46:48.947676 12048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:48.934713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:43503 (local address 127.0.0.1:33526)
0926 16:46:48.934810 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:48.934879 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:48.934949 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:48.936038 (+ 1089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:48.936040 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:48.936193 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:46:48.936255 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:48.936260 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.937685 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.937690 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:48.938456 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:48.938464 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.939056 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.939059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:48.939154 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:48.939584 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:46:48.939598 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:46:48.942154 (+ 2556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:46:48.945552 (+ 3398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:48.945555 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:48.945557 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:48.945802 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:48.946651 (+ 849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:48.946654 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:48.946656 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:48.946749 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:48.947201 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:46:48.947203 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:46:48.947280 (+ 77us) client_negotiation.cc:769] Sending connection context
0926 16:46:48.947369 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 16:46:48.947547 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":3}
Sep 26 16:46:48 dist-test-slave-n1hw krb5kdc[11812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441208, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.125@KRBTEST.COM for kudu/127.21.12.124@KRBTEST.COM
I20260926 16:46:48.948411 12049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:48.934954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33526 (local address 127.21.12.124:43503)
0926 16:46:48.935791 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:48.935799 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:48.935810 (+ 11us) server_negotiation.cc:407] Connection header received
0926 16:46:48.935841 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:48.935844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:48.935880 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:48.935959 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:48.936392 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.937595 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.938595 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.938916 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.939049 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:48.942272 (+ 3223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:48.942275 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:48.942277 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:48.942302 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:48.945451 (+ 3149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:48.946468 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:48.946474 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:48.946477 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:48.946562 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:48.946843 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:48.946846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:48.946849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:48.947080 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:48.947169 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:48.948076 (+ 907us) server_negotiation.cc:299] Negotiation successful
0926 16:46:48.948245 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":781,"thread_start_us":120,"threads_started":1}
I20260926 16:46:48.949471 11893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75670c2011e1454eb6fa99e5110a4ced: Bootstrap starting.
I20260926 16:46:48.952005 11893 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75670c2011e1454eb6fa99e5110a4ced: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:48.952104 11970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:48.941482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.124:43503 (local address 127.0.0.1:33532)
0926 16:46:48.941811 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:48.941836 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:48.941910 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:48.942156 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:48.942159 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:48.942406 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:46:48.942530 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:48.942541 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.945057 (+ 2516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.945061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:48.945918 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:48.945927 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.946530 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.946533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:48.946628 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:48.947088 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:46:48.947102 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:46:48.948532 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:46:48.950545 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:48.950548 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:48.950550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:48.950787 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:48.951131 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:48.951137 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:48.951142 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:48.951238 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:48.951667 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:46:48.951670 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:46:48.951756 (+ 86us) client_negotiation.cc:769] Sending connection context
0926 16:46:48.951834 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 16:46:48.951965 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210}
I20260926 16:46:48.952702 12050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:48.941564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33532 (local address 127.21.12.124:43503)
0926 16:46:48.941896 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:48.941905 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:48.941918 (+ 13us) server_negotiation.cc:407] Connection header received
0926 16:46:48.941986 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:48.941990 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:48.942029 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:48.943965 (+ 1936us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:48.944083 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.944920 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.946066 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.946400 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.946845 (+ 445us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:48.948677 (+ 1832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:48.948681 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:48.948686 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:48.948710 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:48.950421 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:48.950910 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:48.950913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:48.950914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:48.950994 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:48.951364 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:48.951367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:48.951369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:48.951541 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:48.952385 (+ 844us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:48.952456 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 16:46:48.952559 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":119,"threads_started":1}
I20260926 16:46:48.952904 11893 log.cc:824] T 00000000000000000000000000000000 P 75670c2011e1454eb6fa99e5110a4ced: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:48.953130 12044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24b312a595fe4d7e9a4556550e07fcd7: Bootstrap starting.
I20260926 16:46:48.955094 11893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75670c2011e1454eb6fa99e5110a4ced: No bootstrap required, opened a new log
I20260926 16:46:48.955363 12044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24b312a595fe4d7e9a4556550e07fcd7: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:48.955343 11968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba98563b5e644facb8f57d7cd32ec15e: Bootstrap starting.
I20260926 16:46:48.956079 12044 log.cc:824] T 00000000000000000000000000000000 P 24b312a595fe4d7e9a4556550e07fcd7: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:48.957406 21553 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:46:48.957674 11968 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba98563b5e644facb8f57d7cd32ec15e: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:48.958307 12044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24b312a595fe4d7e9a4556550e07fcd7: No bootstrap required, opened a new log
I20260926 16:46:48.958530 11968 log.cc:824] T 00000000000000000000000000000000 P ba98563b5e644facb8f57d7cd32ec15e: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:48.958746 11867 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "75670c2011e1454eb6fa99e5110a4ced"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43320
I20260926 16:46:48.959288 11893 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75670c2011e1454eb6fa99e5110a4ced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75670c2011e1454eb6fa99e5110a4ced" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 41421 } } peers { permanent_uuid: "ba98563b5e644facb8f57d7cd32ec15e" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 42689 } } peers { permanent_uuid: "24b312a595fe4d7e9a4556550e07fcd7" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 43503 } }
I20260926 16:46:48.959499 11893 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75670c2011e1454eb6fa99e5110a4ced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75670c2011e1454eb6fa99e5110a4ced, State: Initialized, Role: FOLLOWER
I20260926 16:46:48.959837 11893 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75670c2011e1454eb6fa99e5110a4ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75670c2011e1454eb6fa99e5110a4ced" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 41421 } } peers { permanent_uuid: "ba98563b5e644facb8f57d7cd32ec15e" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 42689 } } peers { permanent_uuid: "24b312a595fe4d7e9a4556550e07fcd7" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 43503 } }
I20260926 16:46:48.960300 12054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75670c2011e1454eb6fa99e5110a4ced [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75670c2011e1454eb6fa99e5110a4ced" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 41421 } } peers { permanent_uuid: "ba98563b5e644facb8f57d7cd32ec15e" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 42689 } } peers { permanent_uuid: "24b312a595fe4d7e9a4556550e07fcd7" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 43503 } } }
I20260926 16:46:48.960484 12054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75670c2011e1454eb6fa99e5110a4ced [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:46:48.961102 11893 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75670c2011e1454eb6fa99e5110a4ced [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:46:48.961191 11867 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75670c2011e1454eb6fa99e5110a4ced [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 16:46:48.961295 11867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75670c2011e1454eb6fa99e5110a4ced [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:48.962535 12044 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24b312a595fe4d7e9a4556550e07fcd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75670c2011e1454eb6fa99e5110a4ced" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 41421 } } peers { permanent_uuid: "ba98563b5e644facb8f57d7cd32ec15e" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 42689 } } peers { permanent_uuid: "24b312a595fe4d7e9a4556550e07fcd7" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 43503 } }
I20260926 16:46:48.962625 11867 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75670c2011e1454eb6fa99e5110a4ced [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75670c2011e1454eb6fa99e5110a4ced" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 41421 } } peers { permanent_uuid: "ba98563b5e644facb8f57d7cd32ec15e" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 42689 } } peers { permanent_uuid: "24b312a595fe4d7e9a4556550e07fcd7" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 43503 } }
I20260926 16:46:48.962759 12044 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24b312a595fe4d7e9a4556550e07fcd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24b312a595fe4d7e9a4556550e07fcd7, State: Initialized, Role: FOLLOWER
I20260926 16:46:48.963667 11867 leader_election.cc:288] T 00000000000000000000000000000000 P 75670c2011e1454eb6fa99e5110a4ced [CANDIDATE]: Term 1 election: Requested vote from peers ba98563b5e644facb8f57d7cd32ec15e (127.21.12.125:42689), 24b312a595fe4d7e9a4556550e07fcd7 (127.21.12.124:43503)
W20260926 16:46:48.966128 12065 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 75670c2011e1454eb6fa99e5110a4ced: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 16:46:48.966244 12065 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 16:46:48.966881 11968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba98563b5e644facb8f57d7cd32ec15e: No bootstrap required, opened a new log
I20260926 16:46:48.966104 11825 master_runner.cc:186] Error getting master registration for 127.21.12.125:42689: OK, instance_id { permanent_uuid: "ba98563b5e644facb8f57d7cd32ec15e" instance_seqno: 1790441208682717 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 16:46:48.969023 11825 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 16:46:48.963095 12044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24b312a595fe4d7e9a4556550e07fcd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75670c2011e1454eb6fa99e5110a4ced" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 41421 } } peers { permanent_uuid: "ba98563b5e644facb8f57d7cd32ec15e" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 42689 } } peers { permanent_uuid: "24b312a595fe4d7e9a4556550e07fcd7" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 43503 } }
I20260926 16:46:48.969689 12074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24b312a595fe4d7e9a4556550e07fcd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75670c2011e1454eb6fa99e5110a4ced" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 41421 } } peers { permanent_uuid: "ba98563b5e644facb8f57d7cd32ec15e" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 42689 } } peers { permanent_uuid: "24b312a595fe4d7e9a4556550e07fcd7" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 43503 } } }
I20260926 16:46:48.970079 12074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24b312a595fe4d7e9a4556550e07fcd7 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:46:48.970505 12044 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24b312a595fe4d7e9a4556550e07fcd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:46:48.970580 12018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "75670c2011e1454eb6fa99e5110a4ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "24b312a595fe4d7e9a4556550e07fcd7"
I20260926 16:46:48.970777 12018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24b312a595fe4d7e9a4556550e07fcd7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:48.971312 11968 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba98563b5e644facb8f57d7cd32ec15e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75670c2011e1454eb6fa99e5110a4ced" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 41421 } } peers { permanent_uuid: "ba98563b5e644facb8f57d7cd32ec15e" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 42689 } } peers { permanent_uuid: "24b312a595fe4d7e9a4556550e07fcd7" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 43503 } }
I20260926 16:46:48.971509 11968 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba98563b5e644facb8f57d7cd32ec15e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba98563b5e644facb8f57d7cd32ec15e, State: Initialized, Role: FOLLOWER
I20260926 16:46:48.971824 11968 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba98563b5e644facb8f57d7cd32ec15e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75670c2011e1454eb6fa99e5110a4ced" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 41421 } } peers { permanent_uuid: "ba98563b5e644facb8f57d7cd32ec15e" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 42689 } } peers { permanent_uuid: "24b312a595fe4d7e9a4556550e07fcd7" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 43503 } }
I20260926 16:46:48.972240 12018 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 24b312a595fe4d7e9a4556550e07fcd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75670c2011e1454eb6fa99e5110a4ced in term 1.
I20260926 16:46:48.972798 11843 leader_election.cc:302] T 00000000000000000000000000000000 P 75670c2011e1454eb6fa99e5110a4ced [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24b312a595fe4d7e9a4556550e07fcd7, 75670c2011e1454eb6fa99e5110a4ced; no voters:
I20260926 16:46:48.973161 12054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75670c2011e1454eb6fa99e5110a4ced [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:46:48.973369 11968 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba98563b5e644facb8f57d7cd32ec15e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:46:48.973384 12054 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75670c2011e1454eb6fa99e5110a4ced [term 1 LEADER]: Becoming Leader. State: Replica: 75670c2011e1454eb6fa99e5110a4ced, State: Running, Role: LEADER
I20260926 16:46:48.974084 11942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "75670c2011e1454eb6fa99e5110a4ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ba98563b5e644facb8f57d7cd32ec15e"
I20260926 16:46:48.974303 11942 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba98563b5e644facb8f57d7cd32ec15e [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:48.976670 12054 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75670c2011e1454eb6fa99e5110a4ced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75670c2011e1454eb6fa99e5110a4ced" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 41421 } } peers { permanent_uuid: "ba98563b5e644facb8f57d7cd32ec15e" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 42689 } } peers { permanent_uuid: "24b312a595fe4d7e9a4556550e07fcd7" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 43503 } }
I20260926 16:46:48.982439 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:48.966285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43324 (local address 127.21.12.126:41421)
0926 16:46:48.967320 (+ 1035us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:48.967324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:48.967663 (+ 339us) server_negotiation.cc:407] Connection header received
0926 16:46:48.967944 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:48.967947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:48.967993 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:48.968041 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:48.970902 (+ 2861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.971655 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.972825 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.973132 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.973208 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:48.978016 (+ 4808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:48.978022 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:48.978026 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:48.978072 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:48.980356 (+ 2284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:48.980974 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:48.980978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:48.980980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:48.981076 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:48.981388 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:48.981391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:48.981393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:48.981634 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:48.981785 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:48.982153 (+ 368us) server_negotiation.cc:299] Negotiation successful
0926 16:46:48.982284 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":913}
I20260926 16:46:48.982574 12050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:48.966861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33546 (local address 127.21.12.124:43503)
0926 16:46:48.966995 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:48.966999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:48.972748 (+ 5749us) server_negotiation.cc:407] Connection header received
0926 16:46:48.972930 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:48.972933 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:48.972978 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:48.973027 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:48.974034 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.974755 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.975866 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.976167 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.976248 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:48.978458 (+ 2210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:48.978467 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:48.978474 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:48.978510 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:48.980226 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:48.981216 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:48.981221 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:48.981225 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:48.981307 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:48.981612 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:48.981615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:48.981617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:48.981853 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:48.981980 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:48.982300 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 16:46:48.982407 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 16:46:48.982385 11975 master_runner.cc:186] Error getting master registration for 127.21.12.125:42689: OK, instance_id { permanent_uuid: "ba98563b5e644facb8f57d7cd32ec15e" instance_seqno: 1790441208682717 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 16:46:48.985630 11975 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 16:46:48.986263 11942 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ba98563b5e644facb8f57d7cd32ec15e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75670c2011e1454eb6fa99e5110a4ced in term 1.
I20260926 16:46:48.987370 11974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:48.966376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49164 (local address 127.21.12.125:42689)
0926 16:46:48.966477 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:48.966481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:48.969067 (+ 2586us) server_negotiation.cc:407] Connection header received
0926 16:46:48.969855 (+ 788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:48.969858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:48.969903 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:48.969950 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:48.973789 (+ 3839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.974584 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.975760 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.976025 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.976096 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:48.983858 (+ 7762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:48.983862 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:48.983865 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:48.983893 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:48.985390 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:48.985920 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:48.985926 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:48.985929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:48.986025 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:48.986363 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:48.986369 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:48.986373 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:48.986594 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:48.986719 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:48.987067 (+ 348us) server_negotiation.cc:299] Negotiation successful
0926 16:46:48.987200 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 16:46:48.988097 12087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75670c2011e1454eb6fa99e5110a4ced [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75670c2011e1454eb6fa99e5110a4ced. Latest consensus state: current_term: 1 leader_uuid: "75670c2011e1454eb6fa99e5110a4ced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75670c2011e1454eb6fa99e5110a4ced" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 41421 } } peers { permanent_uuid: "ba98563b5e644facb8f57d7cd32ec15e" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 42689 } } peers { permanent_uuid: "24b312a595fe4d7e9a4556550e07fcd7" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 43503 } } }
I20260926 16:46:48.988304 12087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75670c2011e1454eb6fa99e5110a4ced [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:48.989943 12082 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba98563b5e644facb8f57d7cd32ec15e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75670c2011e1454eb6fa99e5110a4ced" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 41421 } } peers { permanent_uuid: "ba98563b5e644facb8f57d7cd32ec15e" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 42689 } } peers { permanent_uuid: "24b312a595fe4d7e9a4556550e07fcd7" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 43503 } } }
I20260926 16:46:48.990082 12082 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba98563b5e644facb8f57d7cd32ec15e [sys.catalog]: This master's current role is: FOLLOWER
W20260926 16:46:48.991631 12099 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ba98563b5e644facb8f57d7cd32ec15e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 16:46:48.991748 12099 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 16:46:48.993142 12100 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260926 16:46:48.993950 12086 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 24b312a595fe4d7e9a4556550e07fcd7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 16:46:48.994023 12086 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 16:46:48.994112 12100 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:46:48.999619 12018 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 24b312a595fe4d7e9a4556550e07fcd7 [term 1 FOLLOWER]: Refusing update from remote peer 75670c2011e1454eb6fa99e5110a4ced: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:46:49.000268 12087 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 75670c2011e1454eb6fa99e5110a4ced [LEADER]: Connected to new peer: Peer: permanent_uuid: "24b312a595fe4d7e9a4556550e07fcd7" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 43503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:46:49.006215 11942 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ba98563b5e644facb8f57d7cd32ec15e [term 1 FOLLOWER]: Refusing update from remote peer 75670c2011e1454eb6fa99e5110a4ced: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:46:49.006603 12074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24b312a595fe4d7e9a4556550e07fcd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75670c2011e1454eb6fa99e5110a4ced. Latest consensus state: current_term: 1 leader_uuid: "75670c2011e1454eb6fa99e5110a4ced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75670c2011e1454eb6fa99e5110a4ced" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 41421 } } peers { permanent_uuid: "ba98563b5e644facb8f57d7cd32ec15e" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 42689 } } peers { permanent_uuid: "24b312a595fe4d7e9a4556550e07fcd7" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 43503 } } }
I20260926 16:46:49.006711 12074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24b312a595fe4d7e9a4556550e07fcd7 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:46:49.006762 12087 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 75670c2011e1454eb6fa99e5110a4ced [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba98563b5e644facb8f57d7cd32ec15e" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 42689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:46:49.010357 12082 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba98563b5e644facb8f57d7cd32ec15e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75670c2011e1454eb6fa99e5110a4ced. Latest consensus state: current_term: 1 leader_uuid: "75670c2011e1454eb6fa99e5110a4ced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75670c2011e1454eb6fa99e5110a4ced" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 41421 } } peers { permanent_uuid: "ba98563b5e644facb8f57d7cd32ec15e" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 42689 } } peers { permanent_uuid: "24b312a595fe4d7e9a4556550e07fcd7" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 43503 } } }
I20260926 16:46:49.010478 12082 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba98563b5e644facb8f57d7cd32ec15e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:46:49.014652 12087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75670c2011e1454eb6fa99e5110a4ced [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "75670c2011e1454eb6fa99e5110a4ced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75670c2011e1454eb6fa99e5110a4ced" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 41421 } } peers { permanent_uuid: "ba98563b5e644facb8f57d7cd32ec15e" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 42689 } } peers { permanent_uuid: "24b312a595fe4d7e9a4556550e07fcd7" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 43503 } } }
I20260926 16:46:49.014771 12087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75670c2011e1454eb6fa99e5110a4ced [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:49.015856 12082 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba98563b5e644facb8f57d7cd32ec15e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "75670c2011e1454eb6fa99e5110a4ced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75670c2011e1454eb6fa99e5110a4ced" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 41421 } } peers { permanent_uuid: "ba98563b5e644facb8f57d7cd32ec15e" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 42689 } } peers { permanent_uuid: "24b312a595fe4d7e9a4556550e07fcd7" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 43503 } } }
I20260926 16:46:49.015962 12082 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba98563b5e644facb8f57d7cd32ec15e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:46:49.016316 12103 mvcc.cc:205] Tried to move back new op lower bound from 7333647192054108160 to 7333647191971979264. Current Snapshot: MvccSnapshot[applied={T|T < 7333647192054108160}]
I20260926 16:46:49.018301 12106 mvcc.cc:205] Tried to move back new op lower bound from 7333647192054108160 to 7333647191971979264. Current Snapshot: MvccSnapshot[applied={T|T < 7333647192054108160}]
I20260926 16:46:49.020605 12054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75670c2011e1454eb6fa99e5110a4ced [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "75670c2011e1454eb6fa99e5110a4ced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75670c2011e1454eb6fa99e5110a4ced" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 41421 } } peers { permanent_uuid: "ba98563b5e644facb8f57d7cd32ec15e" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 42689 } } peers { permanent_uuid: "24b312a595fe4d7e9a4556550e07fcd7" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 43503 } } }
I20260926 16:46:49.020741 12054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75670c2011e1454eb6fa99e5110a4ced [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:49.021797 12074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24b312a595fe4d7e9a4556550e07fcd7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "75670c2011e1454eb6fa99e5110a4ced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75670c2011e1454eb6fa99e5110a4ced" member_type: VOTER last_known_addr { host: "127.21.12.126" port: 41421 } } peers { permanent_uuid: "ba98563b5e644facb8f57d7cd32ec15e" member_type: VOTER last_known_addr { host: "127.21.12.125" port: 42689 } } peers { permanent_uuid: "24b312a595fe4d7e9a4556550e07fcd7" member_type: VOTER last_known_addr { host: "127.21.12.124" port: 43503 } } }
I20260926 16:46:49.021911 12074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24b312a595fe4d7e9a4556550e07fcd7 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:46:49.025810 12100 catalog_manager.cc:1347] Generated new cluster ID: fdaa692751d244dea0f0d3c09e74eb17
I20260926 16:46:49.025876 12100 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:46:49.025972 12104 mvcc.cc:205] Tried to move back new op lower bound from 7333647192054108160 to 7333647191971979264. Current Snapshot: MvccSnapshot[applied={T|T < 7333647192054108160}]
I20260926 16:46:49.050990 12100 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:46:49.052330 12100 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:46:49.074890 12100 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75670c2011e1454eb6fa99e5110a4ced: Generated new TSK 0
I20260926 16:46:49.076108 12100 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 16:46:49.108614 12050 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:46:49.105534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33558 (local address 127.21.12.124:43503)
0926 16:46:49.105660 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:49.105667 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:49.107535 (+ 1868us) server_negotiation.cc:407] Connection header received
0926 16:46:49.107734 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:49.107738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:49.107790 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:49.107842 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:49.108475 (+ 633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33558: BlockingRecv error: recv got EOF from 127.0.0.1:33558 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260926 16:46:49.109988 11897 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:46:49.106320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43336 (local address 127.21.12.126:41421)
0926 16:46:49.106437 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:49.106441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:49.109100 (+ 2659us) server_negotiation.cc:407] Connection header received
0926 16:46:49.109270 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:49.109272 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:49.109321 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:49.109372 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:49.109875 (+ 503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43336: BlockingRecv error: recv got EOF from 127.0.0.1:43336 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260926 16:46:49.111366 11974 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:46:49.106962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49176 (local address 127.21.12.125:42689)
0926 16:46:49.107082 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:49.107086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:49.110444 (+ 3358us) server_negotiation.cc:407] Connection header received
0926 16:46:49.110677 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:49.110680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:49.110731 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:49.110781 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:49.111256 (+ 475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49176: BlockingRecv error: recv got EOF from 127.0.0.1:49176 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260926 16:46:49.115754 11974 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:46:49.114256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49180 (local address 127.21.12.125:42689)
0926 16:46:49.114356 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:49.114360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:49.114865 (+ 505us) server_negotiation.cc:407] Connection header received
0926 16:46:49.115081 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:49.115084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:49.115156 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:49.115206 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:49.115662 (+ 456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49180: BlockingRecv error: recv got EOF from 127.0.0.1:49180 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260926 16:46:49.120565 11974 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:46:49.119206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49194 (local address 127.21.12.125:42689)
0926 16:46:49.119303 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:49.119307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:49.119755 (+ 448us) server_negotiation.cc:407] Connection header received
0926 16:46:49.119920 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:49.119923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:49.119969 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:49.120015 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:49.120475 (+ 460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49194: BlockingRecv error: recv got EOF from 127.0.0.1:49194 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260926 16:46:49.126937 11974 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:46:49.125540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49210 (local address 127.21.12.125:42689)
0926 16:46:49.125637 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:49.125641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:49.126107 (+ 466us) server_negotiation.cc:407] Connection header received
0926 16:46:49.126282 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:49.126285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:49.126333 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:49.126382 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:49.126844 (+ 462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49210: BlockingRecv error: recv got EOF from 127.0.0.1:49210 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260926 16:46:49.134662 11974 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:46:49.133086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49220 (local address 127.21.12.125:42689)
0926 16:46:49.133225 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:49.133229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:49.133705 (+ 476us) server_negotiation.cc:407] Connection header received
0926 16:46:49.133973 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:49.133976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:49.134027 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:49.134086 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:49.134562 (+ 476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49220: BlockingRecv error: recv got EOF from 127.0.0.1:49220 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260926 16:46:49.146871 11974 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:46:49.145015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49228 (local address 127.21.12.125:42689)
0926 16:46:49.145140 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:49.145144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:49.145665 (+ 521us) server_negotiation.cc:407] Connection header received
0926 16:46:49.146012 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:49.146016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:49.146085 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:49.146139 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:49.146732 (+ 593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49228: BlockingRecv error: recv got EOF from 127.0.0.1:49228 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260926 16:46:49.167181 11974 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:46:49.165528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49244 (local address 127.21.12.125:42689)
0926 16:46:49.165649 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:49.165653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:49.166232 (+ 579us) server_negotiation.cc:407] Connection header received
0926 16:46:49.166425 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:49.166429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:49.166480 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:49.166530 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:49.167067 (+ 537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49244: BlockingRecv error: recv got EOF from 127.0.0.1:49244 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260926 16:46:49.202976 11974 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:46:49.201675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49248 (local address 127.21.12.125:42689)
0926 16:46:49.201839 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:49.201846 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:49.202129 (+ 283us) server_negotiation.cc:407] Connection header received
0926 16:46:49.202190 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:49.202193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:49.202242 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:49.202322 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:49.202845 (+ 523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49248: BlockingRecv error: recv got EOF from 127.0.0.1:49248 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260926 16:46:49.270484 11974 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:46:49.269135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49250 (local address 127.21.12.125:42689)
0926 16:46:49.269286 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:49.269292 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:49.269417 (+ 125us) server_negotiation.cc:407] Connection header received
0926 16:46:49.269553 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:49.269559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:49.269610 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:49.269682 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:49.270296 (+ 614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49250: BlockingRecv error: recv got EOF from 127.0.0.1:49250 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260926 16:46:49.401628 11974 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:46:49.400476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48492 (local address 127.21.12.125:42689)
0926 16:46:49.400626 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:49.400631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:49.400792 (+ 161us) server_negotiation.cc:407] Connection header received
0926 16:46:49.400914 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:49.400917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:49.400966 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:49.401065 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:49.401502 (+ 437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48492: BlockingRecv error: recv got EOF from 127.0.0.1:48492 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 16:46:49.660917 11974 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:46:49.659717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48506 (local address 127.21.12.125:42689)
0926 16:46:49.659875 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:49.659879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:49.660034 (+ 155us) server_negotiation.cc:407] Connection header received
0926 16:46:49.660159 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:49.660162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:49.660209 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:49.660283 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:49.660766 (+ 483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48506: BlockingRecv error: recv got EOF from 127.0.0.1:48506 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 16:46:49.993125 12099 catalog_manager.cc:1259] Loaded cluster ID: fdaa692751d244dea0f0d3c09e74eb17
I20260926 16:46:49.993216 12099 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ba98563b5e644facb8f57d7cd32ec15e: loading cluster ID for follower catalog manager: success
I20260926 16:46:49.994935 12099 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ba98563b5e644facb8f57d7cd32ec15e: acquiring CA information for follower catalog manager: success
I20260926 16:46:49.995134 12086 catalog_manager.cc:1259] Loaded cluster ID: fdaa692751d244dea0f0d3c09e74eb17
I20260926 16:46:49.995210 12086 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 24b312a595fe4d7e9a4556550e07fcd7: loading cluster ID for follower catalog manager: success
I20260926 16:46:49.996150 12099 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ba98563b5e644facb8f57d7cd32ec15e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 16:46:49.996774 12086 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 24b312a595fe4d7e9a4556550e07fcd7: acquiring CA information for follower catalog manager: success
I20260926 16:46:49.997778 12086 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 24b312a595fe4d7e9a4556550e07fcd7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 16:46:50.179164 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:50.175084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48510 (local address 127.21.12.125:42689)
0926 16:46:50.175388 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:50.175393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:50.175408 (+ 15us) server_negotiation.cc:407] Connection header received
0926 16:46:50.175574 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:50.175578 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:50.175713 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:50.175827 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:46:50.176103 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:50.176629 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:50.177517 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:50.177796 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:50.177919 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:50.178279 (+ 360us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:46:50.178713 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:46:50.178853 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:50.178894 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 16:46:50.178961 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":88,"threads_started":1}
I20260926 16:46:50.185482 12181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:50.182142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48926 (local address 127.21.12.124:43503)
0926 16:46:50.182365 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:50.182368 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:50.182380 (+ 12us) server_negotiation.cc:407] Connection header received
0926 16:46:50.182462 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:50.182464 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:50.182573 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:50.182665 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:46:50.182889 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:50.183386 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:50.184173 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:50.184389 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:50.184467 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:50.184706 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:46:50.185087 (+ 381us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:46:50.185176 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:50.185237 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 16:46:50.185281 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":67,"threads_started":1}
I20260926 16:46:50.192597 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:50.188570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48524 (local address 127.21.12.125:42689)
0926 16:46:50.188680 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:50.188684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:50.188939 (+ 255us) server_negotiation.cc:407] Connection header received
0926 16:46:50.189214 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:50.189219 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:50.189274 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:50.189357 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:46:50.189607 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:50.190413 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:50.191459 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:50.191765 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:50.191861 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:50.192112 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:46:50.192191 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:46:50.192255 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:50.192404 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 16:46:50.192470 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 16:46:50.193073 12181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:50.188987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48934 (local address 127.21.12.124:43503)
0926 16:46:50.189078 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:50.189082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:50.189194 (+ 112us) server_negotiation.cc:407] Connection header received
0926 16:46:50.189369 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:50.189372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:50.189418 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:50.189489 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:46:50.189778 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:50.190514 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:50.191729 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:50.192007 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:50.192122 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:50.192585 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:46:50.192684 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:46:50.192739 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:50.192911 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 16:46:50.192961 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 16:46:50.193990 12188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:50.188532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35310 (local address 127.21.12.126:41421)
0926 16:46:50.189171 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:50.189178 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:50.189193 (+ 15us) server_negotiation.cc:407] Connection header received
0926 16:46:50.189284 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:50.189290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:50.189441 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:50.189593 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:46:50.189841 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:50.190373 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:50.191738 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:50.192113 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:50.192235 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:50.192596 (+ 361us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:46:50.193554 (+ 958us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:46:50.193649 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:50.193794 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 16:46:50.193869 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":60,"threads_started":1}
I20260926 16:46:50.196863 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 11825
I20260926 16:46:50.203465 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 11900
I20260926 16:46:50.209134 21553 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEZzU7d/build/debug/bin/kudu with pid 11975
2026-09-26T16:46:50Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3075 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6125 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (285265 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST